Model processing method and device and related equipment

Abstract

The invention provides a model processing method and device and related equipment. Controlling the neural network model to operate in the first operation environment and the second operation environment, and comparing whether the output data of the first network node in the two operation environments are consistent or not; if not, the neural network model is controlled to operate again, whether the output data of the second network node in the two operation environments are different or not is compared, and the output data of the second network node in the neural network model influences the output data of the first network node; and generating a model processing result based on the output data comparison result of the second network node, the model processing result indicating the network node with the problem. Therefore, when the neural network model is operated every time, whether the output data of part of network nodes in different operation environments are the same or not can be judged only through comparison, and all network nodes do not need to be compared, so that resource consumption required for positioning problem network nodes in the model can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model processing method, device and related equipment. Background Art

[0002] With the development of artificial intelligence (AI) technology, neural network models are often migrated between different operating environments, such as migrating neural network models between different AI frameworks, or migrating neural network models between different hardware devices. In actual applications, the results obtained by running neural network models in different operating environments may be different. For example, after migrating the neural network model from the deep-learning processing unit (DPU) to the graphics processing unit (GPU), the reasoning accuracy of the neural network model will decrease.

[0003] At present, before migrating a neural network, the neural network model is usually run in different operating environments, and the output data of the neural network model in different operating environments are compared to see if they are consistent. When the output data is inconsistent, technical personnel will perform corresponding repairs on the neural network model to ensure that the reasoning accuracy of the repaired neural network model is consistent with that of the neural network model before migration.

[0004] However, in actual application scenarios, the number of network layers and model parameters included in the neural network model is increasing, such as hundreds of thousands of network layers and tens of millions of model parameters. Therefore, the amount of data generated during the operation of the neural network model is large, which makes it necessary to record the output data of all network layers of the neural network model in different operating environments and compare the consistency between the output data, which will result in excessive resource consumption, including consumption of computing resources and storage resources. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a model processing method for reducing the computing resources and storage resources consumed to locate problems in the model. The present application also provides a corresponding apparatus, a computing device, a computer-readable storage medium, and a computer program product.

[0006] In a first aspect, an embodiment of the present application provides a model processing method, which can be executed by a corresponding model processing device. Specifically, the model processing device controls a neural network model to run in a first operating environment and a second operating environment, and the neural network model has the same input data and random values ​​when running in the first operating environment and the second operating environment, wherein the first operating environment may be the operating environment where the neural network model is located before migration, and the second operating environment may be the operating environment where the neural network model is located after migration; then, the model processing device compares the first output data of the first network node in the neural network model in the first operating environment and the second output data of the first network node in the second operating environment to obtain a first comparison result, and the first comparison result can indicate whether the output data of the first network node in the two operating environments are consistent; when the first comparison result indicates that the first When there is a difference between the output data and the second output data, there may be a problem with the first network node, or there may be a problem with the network node located upstream of the first network node. Therefore, the model processing device can control the neural network model to run in the first operating environment and the second operating environment, and compare the third output data of the second network node in the neural network model in the first operating environment and the fourth output data of the second network node in the second operating environment to obtain a second comparison result, wherein the output data of the second network node in the neural network model affects the output data of the first network node, that is, the second network node is located upstream of the first network node, so as to determine whether there is a problem with the upstream network node; finally, the model processing device generates a model processing result based on the second comparison result, and the model processing result indicates the network node with problems when the neural network model runs in the second operating environment.

[0007] Thus, on the one hand, since each time the neural network model is run in a different operating environment, the model processing device can only compare whether the output data of some network nodes in different operating environments are the same, which can effectively reduce the computing resources and storage resources required to locate the problem in the model compared to the method of recording the output data of all network nodes in different operating environments at one time. On the other hand, when it is determined that the output data of the first network node in the neural network model in the two operating environments are different, the model processing device selects the second network node that affects the output of the first network node from the upstream network node, and determines the network node that causes the problem when the neural network model is running in the second operating environment by re-running the neural network model and comparing whether there is a difference in the output data of the second network node in the two operating environments. In this process, the model processing device does not need to compare the differences between the output data of all network nodes in the neural network model in the two operating environments, that is, for other network nodes whose output data does not affect the output data of the first network node, it is not necessary to compare the differences between their output data in different operating environments, so as to reduce the number of network nodes whose output data need to be compared when locating the network node with problems in the neural network model, that is, it can further reduce the resources consumed by the network node that locates the problem, and can improve the efficiency of problem location.

[0008] In a possible implementation, if the second comparison result indicates that the third output data matches the fourth output data, then the model processing result indicates that there is a problem with the first network node in the neural network model. In this way, after determining that there is no difference between the output data of the upstream network node of the first network node in the two operating environments, it is indicated that the reason for the difference between the output data of the first network node in the two operating environments is that there is a problem with the first network node. Therefore, the model processing device can determine that the first network node is a network node with a problem, thereby locating the network node with a problem.

[0009] In a possible implementation, the second operating environment includes a first processor and a second processor, and the second output data of the first network node in the second operating environment includes a first sub-data and a second sub-data, wherein the first processor is used to execute the neural network model and output the first sub-data, and the second processor is used to execute the neural network model and output the second sub-data; then, when the model processing device compares the first output data with the second output data, it can specifically compare the first sub-data with the second sub-data first, and when the first sub-data is consistent with the second sub-data, it indicates that the output data of the first network node in the same operating environment is consistent, and the model processing device can continue to compare the first output data with the second output data to verify whether the output data of the first network node in different operating environments is consistent. In this way, accidental operating errors of the hardware in the second operating environment can be avoided, resulting in errors in the output data of the first network node in the second operating environment, thereby causing the model processing device to misjudge whether the output data of the first network node in the second operating environment is wrong.

[0010] In one possible implementation, there are differences between the first operating environment and the second operating environment, and the differences may include any of the following: artificial intelligence AI framework, acceleration hardware, different acceleration units in the same acceleration hardware, and hardware platforms. For example, the first operating environment and the second operating environment may be constructed based on different AI frameworks, or based on different types / models of acceleration hardware, or based on different acceleration units in the same acceleration hardware (with different data calculation methods / floating-point operation precision), or based on different hardware platforms (such as based on X86 platform and ARM platform respectively), etc.

[0011] In one possible implementation, the first operating environment includes first acceleration hardware, and the second operating environment includes second acceleration hardware, and the type of the first acceleration hardware is different from the type of the second acceleration hardware. In this way, the model processing device can test whether the output data of the neural network model remains consistent when running on different types of acceleration hardware, and locate the problematic network node when there is inconsistency. Alternatively, the type of the first acceleration hardware is the same as the type of the second acceleration hardware, and the floating-point operation precision of the first acceleration hardware is different from the floating-point operation precision of the second acceleration hardware. In this way, the model processing device can test whether the output data of the neural network model remains consistent when running on acceleration hardware of the same type but different floating-point operation precision, and locate the problematic network node when there is inconsistency, so that after repairing the problematic network node, the neural network model can be migrated to the acceleration hardware with lower floating-point operation precision to run, so as to improve the operating efficiency of the neural network model.

[0012] In a possible implementation, the second comparison result indicates that the third output data does not match the fourth output data. At this time, the second network node may be a network node with a problem, or a network node upstream of the second network node may have a problem. Then, when the model processing device generates the model processing result based on the second comparison result, it may specifically determine the third network node based on the second comparison result, and the output data of the third network node in the neural network model affects the output data of the second network node, that is, the third network node is located upstream of the second network node; then, the model processing device controls the neural network model to run in the first operating environment and the second operating environment, and compares the fifth output data of the third network node in the first operating environment and the sixth output data of the third network node in the second operating environment to obtain a third comparison result, so that when the third comparison result indicates that the fifth output data matches the sixth output data, the model processing device generates a model processing result, and the model processing result indicates that there is a problem with the second network node in the neural network model, or indicates that there is a problem with the network node between the second network node and the third network node. In this way, according to the data transmission characteristics between network nodes, by iteratively executing the neural network model and comparing the differences between the output data, the network node with problems can be gradually located.

[0013] In a possible implementation, the neural network model is executed by the CPU in the first operating environment, and the neural network model is executed by the acceleration hardware in the second operating environment, and the CPU and the acceleration hardware are located in the same computing device. In this way, on a single computing device, it is possible to test whether there is a difference between the output data of the neural network model in the two operating environments, thereby reducing the requirements for the hardware environment in the process of testing the neural network model.

[0014] In a possible implementation, the first network node is a network layer included in the neural network model, or the first network node includes some neurons in a network layer included in the neural network model. In this way, the model processing device can gradually locate the problematic network layer or neuron at the granularity of the network layer or the neuron, so that the technician can repair the problem of the neural network model.

[0015] In a second aspect, the present application provides a model processing device, which includes various modules for executing the model processing method in the first aspect or any possible implementation of the first aspect.

[0016] In a third aspect, the present application provides a computing device, which includes a processor and a memory, the memory stores program instructions, and the processor runs the program instructions to execute the model processing method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computing device, the computing device executes the method described in the first aspect or any one of the implementations of the first aspect.

[0018] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on at least one computing device, enables the at least one computing device to execute the method described in the first aspect or any one of the implementations of the first aspect.

[0019] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided in an embodiment of the present application;

[0021] Figure 2 A flow chart of a model processing method provided in an embodiment of the present application;

[0022] Figure 3 It is a structural diagram of a neural network model;

[0023] Figure 4 A schematic diagram of determining a problematic network node 5 by running the neural network model four times;

[0024] Figure 5 A schematic diagram of the structure of a model processing device provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The scheme in the embodiment provided in this application will be described below in conjunction with the drawings in this application.

[0027] See also Figure 1 , which shows a schematic diagram of the structure of a data processing system. Figure 1 As shown, the data processing system 10 includes multiple processors and model processing devices. For ease of understanding and description, Figure 1 The example including the processor 101, the processor 102, and the model processing device 300 is used as an example for exemplary description.

[0028] Processor 101 (and processor 102) are used to run a neural network model, which may be, for example, one or more of a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), a feedforward neural network (FFNN), or may be another type of neural network model.

[0029] Exemplarily, processor 101 (and processor 102) may be any type of processor, such as a central processing unit (CPU), a deep-learning processing unit (DPU), a GPU, a neural-network processing unit (NPU), or may be other types of processors, without limitation.

[0030] The processor 101 and the processor 102 may form different operating environments. The following provides implementation examples for building different operating environments.

[0031] In the first implementation example, processor 101 and processor 102 may be different types of processors, such as processor 101 may be a DPU and processor 102 may be a GPU. In this way, by running the same neural network model based on different types of processors, the neural network model can be run in different operating environments. At this time, processor 101 and processor 102 may be used as acceleration hardware for offloading computing tasks from the host, and the acceleration hardware may also be referred to as a hardware accelerator, a hardware acceleration chip, etc.

[0032] In the second implementation example, processor 101 and processor 102 may be processors of the same type, but they run the neural network model based on different floating-point calculation precisions. For example, processor 101 may run the neural network model based on 32-bit floating-point calculation precision, that is, the data in the neural network model may be represented by 32 bits; processor 102 runs the neural network model based on 64-bit floating-point calculation precision, etc. At this time, processors based on different floating-point calculation precisions respectively run the same neural network model, so that the neural network model can be run in different operating environments.

[0033] In the third implementation example, the processor 101 and the processor 102 can construct different AI frameworks based on different data processing logics, such as Figure 1 The AI ​​framework 100 and the AI ​​framework 200 shown in the figure. At this time, the intermediate representation (IR) obtained by the processor 101 compiling the neural network model may be different from the IR obtained by the processor 102 compiling the neural network model. Therefore, the neural network model runs in different AI frameworks, that is, the neural network model runs in different operating environments.

[0034] In the fourth implementation example, processor 101 and processor 102 may be processors of different hardware platforms, that is, the instruction set of the architecture supported by processor 101 is different from the instruction set of the architecture supported by processor 102. For example, processor 101 may support the instruction set of the X86 architecture (generally referring to a series of central processing unit instruction set architectures based on Intel 8086 and backward compatible), and processor 102 may support the instruction set of the advanced reduced instructionset computing machines (ARM) architecture, etc. In this way, by running the same neural network model on processors under different hardware platforms, the neural network model can be run in different operating environments.

[0035] In actual application, processor 101 and processor 102 can be deployed on different computing devices. For example, processor 101 and processor 102 can be acceleration hardware integrated on different computing devices, etc. Alternatively, processor 101 and processor 102 can be deployed on the same computing device. For example, processor 101 can be a CPU in a computing device, and processor 102 can be acceleration hardware in a computing device; or, processor 101 and processor 102 can be two different acceleration units in the same acceleration hardware integrated on the same computing device, such as the acceleration hardware can be an AI chip, and processor 101 and processor 102 can be two different types of processor cores on the AI ​​chip, and the data calculation methods of these two processor cores are different. In this way, two different processors can be used on a single computing device to test whether the output data of the neural network model in different operating environments are the same, and locate one or more network nodes in the neural network model that cause different output data.

[0036] The model processing device 300 is used to control the same neural network model to run in different operating environments, and compare the differences between the data generated when the neural network model runs in different operating environments, to determine the network layer or node with problems, so as to solve the problems such as the failure of the neural network model to converge or the reduction of the reasoning accuracy of the neural network model after the neural network model is migrated from one operating environment to another according to the located network layer or node.

[0037] In this process, if the model processing device 300 first saves the output data of all network layers when the neural network model is running in different operating environments, and compares the output data of each network layer in different operating environments layer by layer according to the arrangement order between different network layers, this will require too many storage resources because the total amount of output data of all network layers is too large. In addition, the model processing device 300 will compare the output data of each network layer in two operating environments for each network layer, which will result in a longer time required to locate the network layer with problems in the neural network model, thereby resulting in a higher overall delay in migrating the neural network model between different operating environments.

[0038] Based on this, in the model processing method provided in the present application, the model processing device 300 will first control the neural network model to run in the first operating environment and the second operating environment, the first operating environment is the operating environment constituted by the processor 101, and the second operating environment is the operating environment constituted by the processor 102. The neural network model has the same input data and random values ​​when running in the first operating environment and the second operating environment, and compares the first output data of the first network node in the neural network model in the first operating environment and the second output data of the first network node in the second operating environment, so as to obtain a first comparison result. If the first comparison result indicates that the output data of the first network node in the two operating environments are consistent, it indicates that there is no problem with the first network node and other network nodes upstream of it in the neural network model. At this time, the model processing device 300 can control the neural network model to run again in the two operating environments to continue to locate problems in other network nodes downstream of the first network node. If the first comparison result indicates that the output data of the first network node in the two operating environments are inconsistent, indicating that there is a problem with the first network node or its upstream network node, then the model processing device 300 controls the neural network model to re-run in the first operating environment and the second operating environment, and compares the third output data of the second network node located upstream of the first network node in the first operating environment and the fourth output data in the second operating environment, respectively, to obtain a second comparison result. Thus, the model processing device 300 can generate a model processing result based on the second comparison result, and the model processing result is used to indicate the network node with problems when the neural network model runs in the second operating environment.

[0039] In this way, each time the model processing device 300 runs the neural network model in different operating environments, it only needs to compare whether the output data of a few network nodes in different operating environments are the same. Compared with the method of recording the output data of all network nodes in different operating environments at one time, this can effectively reduce the computing resources and storage resources required to locate problems in the neural network model.

[0040] Moreover, when it is determined that the output data of the first network node in the neural network model in the two operating environments are different, the model processing device 300 selects a second network node that affects the output of the first network node from the upstream network nodes, and determines the network node that is the root cause of the problem when the neural network model is running in the second operating environment by re-running the neural network model and comparing whether there is a difference in the output data of the second network node in the two operating environments. In this process, the model processing device 300 does not need to compare the differences between the output data of all network nodes in the neural network model in the two operating environments, that is, for other network nodes whose output data does not affect the output data of the first network node, the model processing device 300 does not need to compare the differences between their output data in different operating environments, so as to reduce the number of network nodes whose output data need to be compared when locating the network node with problems in the neural network model, that is, it can further reduce the resources consumed by the network node that locates the problem, and can improve the efficiency of problem locating.

[0041] In addition, when the model processing device 300 is implemented in the form of a software tool, the software tool can locate problems for any type of neural network model, thereby decoupling the script file corresponding to the neural network model and reducing the changes to the script file corresponding to the neural network model.

[0042] For example, the above Figure 1 The model processing device 300 shown may be implemented by software or hardware.

[0043] In one example, when implemented by software, the model processing device 300 may be a code running on a computing device including the processor 101 (and the processor 102). Furthermore, when the data processing system 10 includes multiple processors located on different computing devices, the model processing device 300 may include multiple sub-devices, such as sub-device 1 and sub-device 2, wherein sub-device 1 is deployed on the computing device 1 where the processor 101 is located, and is used to control the neural network model to run in the first operating environment formed by the processor 101, and collect the output data of some network nodes; similarly, sub-device 2 is deployed on the computing device 2 where the processor 102 is located, and is used to control the neural network model to run in the second operating environment formed by the processor 102, and collect the output data of the corresponding part of the network nodes. Furthermore, sub-device 1 or sub-device 2 can also compare the output data collected on different computing devices and generate the final model processing result.

[0044] In another example, when implemented by hardware, the model processing device 300 may be hardware with data processing capabilities, etc. Exemplarily, the hardware with data processing capabilities may be a CPU, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, a data processing unit (DPU), etc., or any combination thereof. In addition, the number of hardware with data processing capabilities included in the model processing device 300 may be one or more, and the type of hardware included may be one or more. Specifically, the number and type of the hardware may be set according to the business requirements of the actual application, and this embodiment does not limit this.

[0045] It is worth noting that the above Figure 1 The data processing system 10 shown is only used as an exemplary illustration. In other possible implementations, the data processing system 10 may also include more types of devices, such as a client, etc., which is used to interact with the user, such as obtaining a neural network model to be processed provided by the user; and each operating environment may include multiple processors of the same or different types. Exemplarily, the client, for example, may be an application running on a terminal device on the user side, or may be a web browser provided by the data processing system 10 to the outside. And, for ease of description, Figure 1 The data processing system 10 shown in FIG. 1 only shows a plurality of processors. In actual application, the data processing system 10 may also include a memory, a power supply and other devices required by the plurality of processors to run the neural network model. The specific structure of the data processing system 10 is not limited in the present application.

[0046] Next, various non-limiting specific implementations of the process of processing the neural network model are described in detail.

[0047] See also Figure 2 , is a flow chart of a model processing method in an embodiment of the present application. This method can be applied to the above Figure 1 The data processing system 10 shown in FIG. 1 may also be applied to other possible data processing systems. Figure 1 The data processing system 10 shown is described by way of example. Figure 2 The model processing method shown may specifically include:

[0048] S201: The model processing device 300 controls the neural network model to run in the first operating environment and the second operating environment respectively. The neural network model has the same input data and random values ​​when running in the first operating environment and the second operating environment.

[0049] In this embodiment, before migrating the neural network model between different AI frameworks or different acceleration hardware, the user can first request the application processing system 10 through the client to verify whether the running results of the neural network model in different operating environments are consistent, so as to make targeted adjustments to the network nodes with inconsistent output data in the neural network model, so that after the neural network model is adjusted, the running results of the neural network model in different operating environments remain consistent, so that the user can migrate the adjusted neural network model from the current AI framework / acceleration hardware (that is, the current operating environment) to other AI frameworks / acceleration hardware (other operating environments).

[0050] Among them, the neural network model can include multiple network layers, such as Figure 3 As shown. Each network layer may include one or more nodes, which may also be referred to as neurons, and are the basic computing units in the neural network model. After the nodes in the network layer calculate the corresponding results based on the input data, they will output the results to the nodes in the next network layer based on the data operation order, so that the data is conductive between the nodes in different network layers. Correspondingly, based on the data operation order between the nodes in different network layers, when there is an error in the output data of the node in the upstream network layer, since the output data will be used as the input data of the node in the downstream network layer, the output data calculated by the node in the downstream network layer according to the erroneous input data will also be erroneous. The network node described in this embodiment may specifically be a part of the network layer in the neural network model, such as one of the network layers; or, the network node may specifically be one or more nodes in a network layer, which is not limited to this.

[0051] exist Figure 1In the data processing system 10 shown, the processor 101 can constitute a first operating environment, the processor 102 can constitute a second operating environment, and the model processing device 300 can instruct the processor 101 and the processor 102 to respectively run the same neural network model, so as to control the neural network model to run in the first operating environment and the second operating environment respectively. In this embodiment, the following examples of controlling the neural network model to run in different operating environments are provided.

[0052] In the first implementation example, the user can provide the neural network model to the data processing system 10 through the client provided by the model processing system 10, and send a processing request for the neural network model to the data processing system 10 through the client. The model processing device 300 responds to the request and schedules the neural network model sent by the client to different processors in the data processing system 10, specifically, the neural network model can be scheduled to the processor 101 and the processor 102 respectively, so as to execute the neural network model in different operating environments.

[0053] In the second implementation example, the user can deploy the neural network model on different processors in the data processing system 10 through the client, and send a processing request for the neural network model to the data processing system 10 through the client. The model processing device 300 responds to the request and controls the processor 101 and the processor 102 to start executing the neural network model, thereby realizing that the neural network model runs in different operating environments.

[0054] When controlling the operation of the neural network model, the model processing device 300 can provide the same input data for the neural network models in different operating environments, and set the random values ​​of the neural network models in different operating environments to the same value, so as to eliminate the influence of the differences between the input data and the random values ​​on the output data of each network node in the neural network model.

[0055] Among them, when providing the same input data, the model processing device 300 can provide a set of input data to the neural network model in the first operating environment and the neural network model in the second operating environment, respectively, and trigger the neural network models in different operating environments to start executing the neural network model based on the input data.

[0056] When fixing the random values ​​in the neural network model, the model processing device 300 can adjust the interface for generating random values ​​called by the neural network models in different operating environments to a pre-set interface; the interface can generate the same random value when called by the neural network models in different operating environments. In this way, the neural network models in different operating environments can run based on the same random values. Alternatively, the model processing device 300 can obtain a configuration file provided by the user for the random values ​​in the neural network model, and the configuration file can define fixed values ​​for the variables in the neural network model that need to be assigned random values. In this way, during the operation of the neural network models in different operating environments, the model processing device 300 can set the random values ​​required by each neural network model to the fixed values ​​indicated in the configuration file, so that the neural network models in different operating environments can run based on the same random values.

[0057] S202: The model processing device 300 instructs the processor 101 and the processor 102 to feed back output data of the first network node in the neural network model.

[0058] S203: The processor 101 feeds back the first output data of the first network node in the first operating environment to the model processing device 300, and the processor 102 feeds back the second output data of the first network node in the second operating environment to the model processing device 300.

[0059] S204: The model processing device 300 compares the first output data and the second output data to obtain a first comparison result.

[0060] The first network node may be a network layer in a neural network model, or may be a part of nodes (neurons) in the network layer.

[0061] In this embodiment, when determining whether there is a difference in the running results of the neural network model in two different running environments, the model processing device 300 only compares whether the output data of the same network node in the neural network model in the two running environments are consistent each time the neural network model is run, thereby reducing the storage resources required to cache the output data.

[0062] Therefore, the model processing device 300 can respectively instruct the processor 101 and the processor 102 to feedback the data output by the first network node in the neural network model, and compare the first output data fed back by the processor 101 with the second output data fed back by the processor 102 to obtain a first comparison result.

[0063] If the first comparison result indicates that the first output data is consistent with the second output data, indicating that the output of the first network node in the neural network model can remain consistent when running in different operating environments, then, after migrating the neural network model from the first operating environment to the second operating environment, the first network node will not affect the difference in the operating results of the neural network model in the operating environments before and after the migration.

[0064] If the first comparison result indicates that the first output data is inconsistent with the second output data, indicating that the output data of the first network node when running in different operating environments is inconsistent, this will cause the output difference generated by the first network node after migrating the neural network model from the first operating environment to the second operating environment, resulting in differences in the final output of the neural network model. At this time, the first network node may be the root cause of the different operating results of the neural network model, or it may be that there are errors in the output data of other network nodes upstream of the first network node, resulting in inconsistencies in the output data of the first network node in two different operating environments. Therefore, the model processing device 300 can also further verify whether there are inconsistencies in the output data of other network nodes located upstream of the first network node in the two operating environments by continuing to perform the following steps, so as to determine the root cause of the inconsistent operating results of the neural network model (the root cause is a network node in the neural network model).

[0065] S205: When the first comparison result indicates that there is a difference between the first output data and the second output data, the model processing device 300 controls the neural network model to run again in the first operating environment and the second operating environment.

[0066] It can be understood that, since the processor 101 and the processor 102 only record the output data of the first network node, but not the output data of the remaining network nodes, when executing the neural network model once, the model processing device 300 can control the neural network model to run again in the first operating environment and the second operating environment, so as to collect the output data of other network nodes in the neural network model.

[0067] S206: The model processing device 300 determines a second network node in the neural network model, and instructs the processor 101 and the processor 102 to feed back output data of the second network node, wherein the output data of the second network node in the neural network model affects the output data of the first network node.

[0068] For example, the first network node may be Figure 3 The determined second network node may be Figure 3When there is an error in the output data of network layer B, the output data will be transmitted to network layer D through network layer C, resulting in an error in the output data of network layer D.

[0069] The model processing device 300 may obtain and set a selection strategy for the network node, and select a network node from at least one network node located upstream of the first network node as the second network node.

[0070] Exemplarily, the selection strategy may be, for example, a binary strategy. For example, assume that there are 20 network nodes in the neural network model that are located upstream of the first network node, that is, the output data of these 20 network nodes will affect the output data of the first network node. These 20 network nodes are network nodes 1 to 20, respectively, and network nodes 1 to 20 in the neural network model perform data calculation operations in sequence. Then, the model processing device 300 may select the 10th network node in the middle of the operation order, that is, network node 10, as the second network node.

[0071] Alternatively, the selection strategy may be, for example, an interval selection strategy. Assume that there are 20 network nodes located upstream of the first network node in the neural network model, and these 20 network nodes are network nodes 1 to 20 in the order of data operation. Then, the model processing device 300 may determine that among these 20 network nodes, the network node 16 is spaced 3 network nodes from the first network node (that is, the number of network nodes to be spaced in advance), and determine the network node 16 as the second network node.

[0072] It can be understood that the above-mentioned various selection strategies are only some exemplary explanations. In actual application, the model processing device 300 can also determine the second network node from the upstream of the first network node based on other strategies, and this is not limited.

[0073] S207: The processor 101 feeds back the third output data of the second network node in the first operating environment to the model processing device 300, and the processor 102 feeds back the fourth output data of the second network node in the second operating environment to the model processing device 300.

[0074] S208: The model processing device 300 compares the third output data and the fourth output data to obtain a second comparison result.

[0075] S209: The model processing device 300 generates a model processing result based on the second comparison result, and the model processing result indicates the network nodes that have problems when the neural network model runs in the second running environment.

[0076] The following describes two different situations indicated by the second comparison result.

[0077] Case 1: The second comparison result indicates that the third output data is consistent with the fourth output data. That is, the output of the second network node in the neural network model can be consistent when running in different operating environments, which indicates that there is a problem with the network node between the second network node and the first network node, and the output data error of the network node causes the output data of the first network node in the second operating environment to be wrong, or the problematic network node is the first network node.

[0078] At this time, the model processing device 300 can further determine whether the second network node is the upstream network node closest to the first network node. If so, the model processing device 300 can determine that the first network node is a problematic network node, and generate a corresponding model processing result to indicate that the problematic network node is the first network node. If there is at least one network node between the second network node and the first network node, the model processing device 300 can select a new network node from the at least one network node in the interval as the third network node, such as determining the third network node through a dichotomy strategy or an interval selection strategy. Then, the model processing device 300 refers to the implementation process described in the above steps S205 to S208 to detect whether the output data of the third network node in the two operating environments are consistent, and thereby determine the problematic network node. Repeat the above process, the model processing device 300 gradually narrows the scope of the problematic network node by running the neural network model multiple times in different operating environments, and finally determines the problematic network node.

[0079] For example, assuming that the first network node is Figure 3 The network layer D in the second network node is Figure 3 When the second comparison result indicates that the output data of network layer B in the two operating environments are consistent, the model processing device 300 can determine that the problematic network node is network layer C or network layer D. Then, the model processing device 300 can control the neural network model in different operating environments again, and collect whether the output data of network layer C (that is, the third network node mentioned above) in different operating environments are consistent. If they are consistent, the model processing device 300 can determine that network layer D is the problematic network node; and if they are inconsistent, the model processing device 300 can determine that network layer C is the problematic network node.

[0080] Case 2: The second comparison result indicates that the third output data is inconsistent with the fourth output data. That is, the output data of the second network node when running in different operating environments is inconsistent. At this time, the root cause of the different operating results of the neural network model in the two operating environments may be the second network node, or may be a network node located upstream of the second network node. Therefore, the model processing device 300 can be located in at least one network node upstream of the second network node, and a new network node can be selected as the third network node, such as the third network node can be determined by a dichotomy strategy or an interval selection strategy. Then, the model processing device 300 can refer to the implementation process described in the above steps S205 to S208, compare whether the fifth output data of the third network node in the first operating environment is consistent with the sixth output data in the second operating environment, and obtain the corresponding third comparison result, so as to further determine the network node with problems according to the third comparison result. For example, when the third comparison result indicates that the fifth output data matches the sixth output data, the model processing device 300 can generate a model processing result, and the model processing result indicates that there is a problem with the second network node in the neural network model, or indicates that there is a problem with the network node between the second network node and the third network node. In actual application, by repeatedly executing the above process, the model processing device 300 can finally determine the one or more network nodes with problems, and generate corresponding model processing results to indicate the one or more network nodes with problems.

[0081] For example, assuming that the first network node is Figure 3 The network layer D in the second network node is Figure 3 When the second comparison result indicates that the output data of network layer B in the two operating environments are inconsistent, the model processing device 300 can determine that the problematic network node is network layer A or network layer B. Then, the model processing device 300 can control the neural network model in different operating environments again, and collect whether the output data of network layer A (that is, the third network node mentioned above) in different operating environments are consistent. If they are consistent, the model processing device 300 can determine that network layer B is the problematic network node; and if they are inconsistent, the model processing device 300 can determine that network layer A is the problematic network node.

[0082] In this way, when the model processing device 300 is determining problematic network nodes, it only needs to store the output data of one network node in different operating environments each time the neural network model is run, without having to store the output data of all network nodes, which can effectively reduce the occupancy of storage resources.

[0083] Moreover, the model processing device 300 determines the problematic network node from the network nodes upstream of the first network node based on the above-mentioned binary search strategy or interval selection strategy, which makes it unnecessary for the model processing device 300 to compare the differences between the output data of all network nodes in different operating environments, thereby reducing the number of network nodes that need to be compared, which not only improves the efficiency of locating problematic network nodes, but also saves the consumption of computing resources.

[0084] For example, suppose the neural network model includes Figure 4 The eight network nodes shown are network nodes 1 to 8, respectively. The model processing device 300 can control the neural network model to run for the first time in the first operating environment and the second operating environment, and collect the output data of network node 8 (that is, the first network node mentioned above), and determine that the output data of network node 8 in the two operating environments are inconsistent by comparing the output data. Then, the model processing device 300 can select network node 4 as the second network node from the upstream network nodes 1 to 7 through a binary strategy, and control the neural network model to run for the second time in the first operating environment and the second operating environment. Then, the model processing device 300 collects the output data of network node 4, and determines that the output data of network node 4 in the two operating environments are inconsistent by comparing the output data. Similarly, the model processing device 300 can select network node 6 from network nodes 4 to 8 according to a binary strategy, and control the neural network model to run for the third time in the first operating environment and the second operating environment, collect the output data of network node 6, and determine that the output data of network node 6 in the two operating environments are inconsistent by comparing the output data. Then, the model processing device 300 can select network node 5 from network node 4 to network node 6 according to the dichotomy strategy, and control the neural network model to run for the fourth time in the first operating environment and the second operating environment, collect the output data of network node 6, and determine that the output data of network node 6 in the two operating environments are inconsistent by comparing the output data. In this way, the model processing device 300 can determine that network node 5 is a problematic network node. In this process, if Figure 4 As shown, the model processing device 300 only needs to compare whether the output data of the four network nodes 4, 5, 6, and 8 in the two operating environments are consistent, and it can determine that network node 5 is the network node that causes problems when the neural network model runs in the second operating environment. There is no need to compare network nodes 1, 2, 3, and 7, thereby effectively reducing the number of network nodes whose output data need to be compared and reducing computing resource consumption.

[0085] Furthermore, the model processing device 300 can also output the generated model processing result, such as outputting the model processing result to the client so that the client can present the model processing result to the user. In this way, the user can perform corresponding repairs on the neural network model according to the presented model processing result. For example, assuming that the accuracy of the output data of the network node 5 of the neural network model is reduced, resulting in errors in the operating results of the neural network model in the second operating environment, the user can perform accuracy compensation on the output data of the network node 5, etc., so that the accuracy of the data output by the network node 5 when running in the second operating environment is consistent with the accuracy of the data output by the network node 5 when running in the first operating environment.

[0086] In actual application, the network nodes that cause errors in the running results of the neural network model in the second running environment may include multiple network nodes. Therefore, after the user repairs the corresponding network nodes according to the model processing results, the user can provide the adjusted neural network model to the data processing system 10 through the client, and request the data processing system 10 to verify again whether the running results of the adjusted neural network model in the two running environments are consistent. When it is determined that the running results are inconsistent, the model processing device 300 can detect other network nodes that cause errors in the running results of the neural network model in the second running environment based on the implementation process described in steps S201 to S209 above, and generate and output the model processing results used to indicate the network nodes to indicate the other network nodes, so that the user can continue to adjust the neural network model according to the newly generated model processing results. In this way, by iterating the above process, the running results of the neural network model in the second running environment are finally obtained. The running results in the first running environment are consistent, and at this time, the user can migrate the neural network model from the first running environment to the second running environment.

[0087] In actual application, different input data may also affect the consistency of the running results of the neural network model in the two running environments. For example, in the process of running the neural network model in different running environments, when the input data is A, the running results of the neural network model in different running environments may be consistent, and when the input data is B, there may be differences in the running results of the neural network model in different running environments. Therefore, in a further possible implementation, the user can provide multiple sets of input data to the data processing system 10, and the model processing device 300 can use each set of input data to run the neural network model in the first running environment and the second running environment, respectively, and verify whether the running results of the neural network model in different running environments are consistent. If they are consistent, the model processing device 300 uses the next set of input data to control the neural network model to continue running in different running environments, and verify whether the running results are consistent.

[0088] When there is a set of input data that causes the running results of the neural network model in different running environments to be inconsistent, the model processing device 300 can use the set of input data to determine the network node that causes the running result of the neural network model in the second running environment to be erroneous based on the implementation process described in the above steps S201 to S209. In the process of controlling the neural network model to run repeatedly in different running environments, the model processing device 300 can provide the same set of input data to the neural network model each time. Alternatively, during the first operation of the neural network model based on the set of input data, the model processing device 300 can cache the intermediate data obtained by the processor 101 after preprocessing the set of input data, and the intermediate data is the input data of the first network layer in the neural network model. In this way, each time the neural network model is repeatedly run based on the set of input data, the model processing device 300 can input the cached intermediate data into the first network layer of the neural network model in each operating environment. In this way, when the neural network model is repeatedly run for the second time and subsequently, the processor 101 and the processor 102 no longer need to perform the operation of preprocessing the input data (that is, there is no need to perform the process of preprocessing the input data to obtain the input of the first network layer), thereby further reducing the computing resources required to locate the network nodes with problems.

[0089] In addition, the processor in the operating environment may also generate data processing errors when executing the neural network model, which may affect the output data of some network nodes in the neural network model, such as causing errors in the output data of some network nodes. Therefore, in a further possible implementation, the second operating environment may include processor 3 ( Figure 1(not shown), the processor 3 is the same as the processor 102. Then, when the model processing device 300 controls the neural network model to run in the second operating environment, it can specifically instruct the processor 102 and the processor 3 to execute the neural network model respectively, and the input data and random values ​​used by each processor when executing the neural network model are the same. Then, the processor 102 can feedback the output data of the first network node (and other network nodes) in the process of executing the neural network model to the model processing device 300, hereinafter referred to as the first sub-data, and the processor 3 will also feedback the output data of the first network node in the process of executing the neural network model to the model processing device 300, hereinafter referred to as the second sub-data. In this way, before comparing the output data of the first network node in the two operating environments, the model processing device 300 can first compare whether the first sub-data and the second sub-data in the second operating environment are consistent. When the first sub-data and the second sub-data are consistent, the model processing device 300 then compares the first output data of the first network node in the first operating environment with the second output data of the first network node in the second operating environment (that is, the first sub-data / second sub-data) to determine whether the output data of the first network node in the second operating environment is wrong. When the first sub-data is inconsistent with the second sub-data, the model processing device 300 can re-instruct the processor 102 and the processor 3 in the second operating environment to re-execute the neural network model and feed back the output data of the first network node. In this way, accidental operating errors of the hardware in the second operating environment can be avoided, which may cause the model processing device 300 to misjudge whether the output data of the first network node in the second operating environment is erroneous. When the second operating environment includes a larger number of processors, and the multiple processors all execute the neural network model, the multiple processors will feed back the output data of the first network node, then the second output data received by the model processing device 300 can be the output data determined after voting on the output data fed back by the multiple processors.

[0090] It is worth noting that the above Figure 1 The model processing method shown is only an exemplary description and is not intended to be limiting. Figure 1 More method steps may be added on the basis of the method steps shown, or the execution order of the above steps S201 to S209 may be changed, or some of the above method steps may be replaced accordingly.

[0091] In addition, the above Figure 1The method embodiment shown can be applied to migrate neural network models between different AI frameworks or different acceleration hardware, and in other possible application scenarios, the first operating environment and the second operating environment can also be constructed using the same type of acceleration hardware (such as GPU, etc.) but with different floating-point arithmetic precisions. In this way, the model processing device 300 can verify whether the execution results obtained when the neural network model is executed on two acceleration hardware with different floating-point arithmetic precisions are consistent based on the above method, and further determine some network nodes that cause inconsistent execution results. In this way, after the user repairs the problem for this part of the network nodes, the neural network model can be migrated to the acceleration hardware with lower floating-point arithmetic precision for execution, thereby improving the execution efficiency of the neural network model (that is, the lower the floating-point arithmetic precision, the higher the execution efficiency of the neural network model).

[0092] Above Figure 2 In the illustrated embodiment, the model processing device 300 involved in the model processing process may be software configured on a computing device, and by running the software on the computing device, the computing device may implement the functions of the above-mentioned model processing device 300. Below, based on the perspective of hardware device implementation, the model processing device 300 involved in the model processing process is introduced in detail.

[0093] See also Figure 5 , showing a schematic diagram of the structure of a model processing device, Figure 5 The model processing device 500 shown corresponds to the above Figure 2 The model processing apparatus 300 in the illustrated embodiment.

[0094] like Figure 5 As shown, the model processing device 500 includes:

[0095] A control module 501 is used to control the neural network model to run in the first operating environment and the second operating environment, and the neural network model has the same input data and random values ​​when running in the first operating environment and the second operating environment;

[0096] A comparison module 502, used to compare first output data of a first network node in a first operating environment and second output data of the first network node in a second operating environment in a neural network model to obtain a first comparison result;

[0097] The control module 501 is further configured to control the neural network model to run in the first operating environment and the second operating environment when the first comparison result indicates that there is a difference between the first output data and the second output data;

[0098] The comparison module 502 is further used to compare the third output data of the second network node in the neural network model in the first operating environment and the fourth output data of the second network node in the second operating environment to obtain a second comparison result, where the output data of the second network node in the neural network model affects the output data of the first network node;

[0099] The generation module 503 is used to generate a model processing result based on the second comparison result, and the model processing result indicates the network nodes where there are problems when the neural network model runs in the second running environment.

[0100] In a possible implementation, if the second comparison result indicates that the third output data matches the fourth output data, then the model processing result indicates that there is a problem with the first network node in the neural network model.

[0101] In a possible implementation, the second operating environment includes a first processor and a second processor, the second output data includes first sub-data and second sub-data, the first processor is used to execute the neural network model and output the first sub-data, and the second processor is used to execute the neural network model and output the second sub-data;

[0102] The comparison module 502 is specifically used for:

[0103] comparing the first sub-data and the second sub-data;

[0104] When the first sub-data is consistent with the second sub-data, the first output data is compared with the second output data.

[0105] In one possible implementation, there are differences between the first operating environment and the second operating environment, and the differences include any of the following: artificial intelligence AI framework, acceleration hardware, different acceleration units in the same acceleration hardware, and hardware platform.

[0106] In a possible implementation, the first operating environment includes first acceleration hardware, and the second operating environment includes second acceleration hardware;

[0107] The type of the first acceleration hardware is different from the type of the second acceleration hardware;

[0108] Alternatively, the type of the first acceleration hardware is the same as the type of the second acceleration hardware, and the floating-point operation precision of the first acceleration hardware is different from the floating-point operation precision of the second acceleration hardware.

[0109] In a possible implementation, the first network node is a network layer included in the neural network model, or the first network node includes some neurons in a network layer included in the neural network model.

[0110] In one possible implementation, the second comparison result indicates that the third output data does not match the fourth output data;

[0111] The control module 501 is further used to determine, based on the second comparison result, a third network node, wherein output data of the third network node in the neural network model affects output data of the second network node; and control the neural network model to run in the first operating environment and the second operating environment;

[0112] The comparison module 502 is further used to compare the fifth output data of the third network node in the first operating environment and the sixth output data of the third network node in the second operating environment to obtain a third comparison result;

[0113] The generation module 503 is specifically used to generate a model processing result when the third comparison result indicates that the fifth output data matches the sixth output data, and the model processing result indicates that there is a problem with the second network node in the neural network model, or indicates that there is a problem with the network node between the second network node and the third network node.

[0114] In one possible implementation, the neural network model is executed by a CPU in a first operating environment, and the neural network model is executed by acceleration hardware in a second operating environment, and the CPU and the acceleration hardware are located in the same computing device.

[0115] because Figure 5 The model processing device 500 shown corresponds to Figure 2 The model processing device 300 in the method embodiment shown in FIG. Figure 5 The specific implementation method of the model processing device 500 shown and its technical effects can be found in the relevant descriptions of the aforementioned embodiments, and will not be elaborated here.

[0116] Figure 6 The schematic diagram of the structure of a computing device is shown. The above-mentioned model processing device 300 can be deployed on the computing device. The computing device can be any computing device in the computing device cluster, and can be specifically used to implement the above-mentioned Figure 2 The functions of the model processing device 300 in the illustrated embodiment.

[0117] like Figure 6As shown, computing device 600 includes processor 610, memory 620, communication interface 630 and bus 640. Processor 610, memory 620 and communication interface 630 communicate through bus 640. Bus 640 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The communication interface 630 is used to communicate with the outside, for example, to receive a neural network model to be migrated submitted by a user.

[0118] Among them, the processor 610 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits. The processor 610 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, the functions of the model processing device 300 can be completed by the hardware integrated logic circuit or software instructions in the processor 610. The processor 610 can also be a general processor, a digital signal process (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. Among them, the general processor can be a microprocessor or the processor can also be any conventional processor, etc., and the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 620 , and the processor 610 reads the information in the memory 620 and completes part or all of the functions in the model processing device 300 in combination with its hardware.

[0119] The memory 620 may include a volatile memory, such as a random access memory (RAM), or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD, or a SSD.

[0120] The memory 620 stores executable codes, and the processor 610 executes the executable codes to execute the method executed by the aforementioned model processing device 300 .

[0121] Specifically, in implementing Figure 2 In the case of the embodiment shown, and Figure 2 In the case where the model processing device 300 described in the illustrated embodiment is implemented by software, the execution Figure 2 The software or program code required for the functions of the model processing device 300 is stored in the memory 620. The interaction between the model processing device 300 and other devices is realized through the communication interface 630. The processor is used to execute the instructions in the memory 620 to implement the method executed by the model processing device 300.

[0122] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on one or more computing devices, the one or more computing devices execute the method executed by the model processing device 300 of the above embodiment.

[0123] In addition, the embodiment of the present application further provides a computer program product, and when the computer program product is executed by one or more computing devices, the one or more computing devices execute any of the aforementioned model processing methods. The computer program product may be a software installation package, and when any of the aforementioned model processing methods is required, the computer program product may be downloaded and executed on a computer.

[0124] It should also be noted that the embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the embodiment drawings provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0125] The terms "first", "second", etc. in the specification, claims and the above drawings of this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0126] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0127] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0128] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A model processing method, characterized in that: The method comprises: Controlling the neural network model to run in a first operating environment and a second operating environment, wherein the neural network model has the same input data and random values ​​when running in the first operating environment and the second operating environment; Comparing first output data of a first network node in the neural network model in the first operating environment and second output data of the first network node in the second operating environment to obtain a first comparison result; When the first comparison result indicates that there is a difference between the first output data and the second output data, controlling the neural network model to run in the first operating environment and the second operating environment; Comparing the third output data of the second network node in the neural network model in the first operating environment and the fourth output data of the second network node in the second operating environment to obtain a second comparison result, where the output data of the second network node in the neural network model affects the output data of the first network node; Based on the second comparison result, a model processing result is generated, and the model processing result indicates network nodes where problems exist when the neural network model is running in the second operating environment.

2. The method according to claim 1, characterized in that If the second comparison result indicates that the third output data matches the fourth output data, then the model processing result indicates that there is a problem with the first network node in the neural network model.

3. The method according to claim 1 or 2, characterized in that: The second operating environment includes a first processor and a second processor, the second output data includes first sub-data and second sub-data, the first processor is used to execute the neural network model and output the first sub-data, and the second processor is used to execute the neural network model and output the second sub-data; The comparing first output data of the first network node in the neural network model in the first operating environment and second output data of the first network node in the second operating environment includes: comparing the first sub-data and the second sub-data; When the first sub-data is consistent with the second sub-data, the first output data is compared with the second output data.

4. The method according to any one of claims 1 to 3, characterized in that: There are differences between the first operating environment and the second operating environment, and the differences include any one of the following: artificial intelligence AI framework, acceleration hardware, different acceleration units in the same acceleration hardware, and hardware platform.

5. The method according to claim 4, characterized in that The first operating environment includes first acceleration hardware, and the second operating environment includes second acceleration hardware; The type of the first acceleration hardware is different from the type of the second acceleration hardware; Alternatively, the type of the first acceleration hardware is the same as the type of the second acceleration hardware, and the floating-point operation precision of the first acceleration hardware is different from the floating-point operation precision of the second acceleration hardware.

6. The method according to any one of claims 1 to 5, characterized in that: The second comparison result indicates that the third output data does not match the fourth output data, and generating a model processing result based on the second comparison result includes: Based on the second comparison result, determining a third network node, wherein output data of the third network node in the neural network model affects output data of the second network node; Controlling the neural network model to run in the first operating environment and the second operating environment; comparing fifth output data of the third network node in the first operating environment and sixth output data of the third network node in the second operating environment to obtain a third comparison result; When the third comparison result indicates that the fifth output data matches the sixth output data, the model processing result is generated, and the model processing result indicates that there is a problem with the second network node in the neural network model, or indicates that there is a problem with the network node between the second network node and the third network node.

7. The method according to any one of claims 1 to 6, characterized in that: The neural network model is executed by a central processing unit (CPU) in the first operating environment, and the neural network model is executed by acceleration hardware in the second operating environment, and the CPU and the acceleration hardware are located in the same computing device.

8. The method according to any one of claims 1 to 7, characterized in that: The first network node is a network layer included in the neural network model, or the first network node includes some neurons in a network layer included in the neural network model.

9. A model processing device, characterized in that: The device comprises: A control module, used for controlling the neural network model to run in a first operating environment and a second operating environment, wherein the neural network model has the same input data and random values ​​when running in the first operating environment and the second operating environment; A comparison module, used for comparing first output data of a first network node in the neural network model in the first operating environment and second output data of the first network node in the second operating environment to obtain a first comparison result; The control module is further configured to control the neural network model to run in the first operating environment and the second operating environment when the first comparison result indicates that there is a difference between the first output data and the second output data; The comparison module is further used to compare the third output data of the second network node in the neural network model in the first operating environment and the fourth output data of the second network node in the second operating environment to obtain a second comparison result, where the output data of the second network node in the neural network model affects the output data of the first network node; A generation module is used to generate a model processing result based on the second comparison result, and the model processing result indicates the network nodes where there are problems when the neural network model is running in the second operating environment.

10. The device according to claim 9, characterized in that If the second comparison result indicates that the third output data matches the fourth output data, then the model processing result indicates that there is a problem with the first network node in the neural network model.

11. The device according to claim 9 or 10, characterized in that The second operating environment includes a first processor and a second processor, the second output data includes first sub-data and second sub-data, the first processor is used to execute the neural network model and output the first sub-data, and the second processor is used to execute the neural network model and output the second sub-data; The comparison module is used for: comparing the first sub-data and the second sub-data; When the first sub-data is consistent with the second sub-data, the first output data is compared with the second output data.

12. The device according to any one of claims 9 to 11, characterized in that There are differences between the first operating environment and the second operating environment, and the differences include any one of the following: artificial intelligence AI framework, acceleration hardware, different acceleration units in the same acceleration hardware, and hardware platform.

13. The device according to claim 12, characterized in that The first operating environment includes first acceleration hardware, and the second operating environment includes second acceleration hardware; The type of the first acceleration hardware is different from the type of the second acceleration hardware; Alternatively, the type of the first acceleration hardware is the same as the type of the second acceleration hardware, and the floating-point operation precision of the first acceleration hardware is different from the floating-point operation precision of the second acceleration hardware.

14. The device according to any one of claims 1 to 7, characterized in that The first network node is a network layer included in the neural network model, or the first network node includes some neurons in a network layer included in the neural network model.

15. A computing device, characterized in that: The computing device comprises a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method according to any one of claims 1 to 8.

16. A computer program product comprising instructions, characterized in that When the method is executed on at least one computing device, the at least one computing device is caused to execute the method according to any one of claims 1 to 8.

Images