Model anomaly evaluation method and device, medium and program product

Abstract

The invention discloses a model anomaly evaluation method and device, a medium and a program product, and relates to the field of deep learning. The method comprises the steps of obtaining a to-be-evaluated neural network model constructed by a deep learning framework, and dividing a computational graph of the to-be-evaluated neural network model into a to-be-analyzed module set; determining calculation amount statistics associated data of the double-level application program interface of the to-be-analyzed module set; wherein the double-level application program interface comprises an external interaction interface and a computational graph operator interface; calculating the calculation amount of an external interaction interface and the calculation amount of a graph operator interface according to the calculation amount statistics associated data; and according to the calculation amount of the external interaction interface and the calculation amount of the calculation graph operator interface, automatically performing model anomaly evaluation. According to the technical scheme of the embodiment of the invention, the dependence of model calculation amount evaluation and framework docking abnormity positioning on manpower can be broken through, and the identification efficiency of the abnormal model is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method, device, medium, and program product for evaluating model anomalies. Background Technology

[0002] In deep learning frameworks, during the transformation of the model from external interaction interfaces (such as Python APIs) to underlying computation graph operators, the framework performs a series of optimizations (such as operator fusion, constant folding, etc.) to improve computational efficiency.

[0003] When creating and delivering a model based on an optimized deep learning framework, it's crucial to quickly determine the model's computational cost within a certain error range, along with a corresponding evaluation method, before actual delivery. Typically, this involves extensive code review and evaluation of computational costs by multiple experienced technical personnel, resulting in high labor costs.

[0004] During the development of the delivery model, the framework takes over the operator and algorithm layers. When problems occur with the interface operators, it can lead to inconsistencies between the external interaction interface layer and the underlying operator implementation, making debugging difficult. Summary of the Invention

[0005] This invention provides a method, device, medium, and program product for model anomaly assessment, in order to solve the problems of high manual costs and low model assessment efficiency in model computation assessment and framework docking anomaly localization.

[0006] According to one aspect of the present invention, a method for evaluating model anomalies is provided, comprising:

[0007] Obtain the neural network model to be evaluated constructed by the deep learning framework, and divide the computation graph of the neural network model to be evaluated into a set of modules to be analyzed;

[0008] Determine the computational complexity statistics of the two-tier application programming interfaces (APIs) of the module set to be analyzed; wherein the two-tier APIs include external interaction interfaces and computation graph operator interfaces;

[0009] Based on the computational workload statistics and related data, calculate the computational workload of the external interaction interface and the computational workload of the computation graph operator interface;

[0010] Based on the computational load of the external interaction interface and the computational load of the computation graph operator interface, the model anomaly assessment is performed automatically.

[0011] According to another aspect of the present invention, a model anomaly evaluation apparatus is provided, comprising:

[0012] The computation graph partitioning module is used to obtain the neural network model to be evaluated constructed by the deep learning framework and partition the computation graph of the neural network model to be evaluated into a set of modules to be analyzed.

[0013] The computational load statistics correlation data determination module is used to determine the computational load statistics correlation data of the two-level application programming interfaces of the set of modules to be analyzed; wherein, the two-level application programming interfaces include external interaction interfaces and computation graph operator interfaces;

[0014] The computational load determination module is used to statistically analyze related data based on the computational load, and to calculate the computational load of external interaction interfaces and computational graph operator interfaces.

[0015] The model anomaly assessment module is used to automatically assess model anomalies based on the computational load of the external interaction interface and the computational load of the computation graph operator interface.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory that is communicatively connected to at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the model anomaly evaluation method of any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the model anomaly evaluation method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the model anomaly evaluation method according to any embodiment of the present invention.

[0022] The technical solution of this invention obtains the neural network model to be evaluated constructed by a deep learning framework, divides the computation graph of the neural network model to be evaluated into a set of modules to be analyzed, thereby determining the computational quantity statistical correlation data of the two-level application interface of the set of modules to be analyzed. Then, based on the computational quantity statistical correlation data, the computational quantity of the external interaction interface and the computational quantity of the computation graph operator interface are calculated. Furthermore, based on the computational quantity of the external interaction interface and the computational quantity of the computation graph operator interface, model anomaly evaluation is automatically performed. This can realize automatic evaluation of model computational quantity, and by comparing the computational quantity of the two-level interface, framework docking anomalies can be quickly located, thereby improving the development efficiency and running performance of deep learning models. It solves the problems of high manual cost and low model evaluation efficiency in model computational quantity evaluation and framework docking anomaly location, and can break the dependence on manual labor in model computational quantity evaluation and framework docking anomaly location, greatly improving the identification efficiency of abnormal models.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a model anomaly evaluation method provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of a model anomaly evaluation method provided in Embodiment 2 of the present invention;

[0027] Figure 3 This is a schematic diagram of the overall process of a model anomaly assessment method provided in Embodiment 2 of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a model anomaly evaluation device provided in Embodiment 3 of the present invention;

[0029] Figure 5 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a model anomaly assessment method provided in Embodiment 1 of the present invention. This embodiment is applicable to the automatic assessment of model computational load and the automatic location of framework docking anomalies. The method can be executed by a model anomaly assessment device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0034] Step 110: Obtain the neural network model to be evaluated constructed by the deep learning framework, and divide the computation graph of the neural network model to be evaluated into a set of modules to be analyzed.

[0035] The neural network model to be evaluated can be a model built using a deep learning framework that requires monitoring of computational complexity. Deep learning frameworks can include, but are not limited to, PyTorch, TensorFlow, and ONNX. The set of modules to be analyzed can be the set of modules obtained by partitioning the computational graph of the neural network model, i.e., the module set.

[0036] In this embodiment of the invention, the neural network model to be evaluated, constructed based on the required deep learning framework, can be determined first. Then, according to the logical semantics and input-output shape, the computation graph of the neural network model to be evaluated is divided to obtain a set of modules to be analyzed corresponding to the neural network model to be evaluated, that is, a set consisting of multiple modules into which the computation graph of the neural network model to be evaluated is divided.

[0037] Step 120: Determine the computational complexity statistics of the two-level application programming interfaces of the module set to be analyzed.

[0038] The two-tiered application programming interface (API) can be the interface of the computation graph corresponding to the set of modules to be analyzed at different levels. The two-tiered API can include external interaction interfaces and computation graph operator interfaces. External interaction interfaces can be interfaces that use programming languages ​​to interact with external services, libraries, or frameworks, i.e., high-level programming interfaces for users. Computation graph operator interfaces can be operator interfaces in the computation graph, i.e., interfaces for the underlying computational core, used to directly manipulate tensors or hardware instructions. External interaction interfaces can include, but are not limited to, Python APIs. For example, one external interaction interface can correspond to one or more computation graph operator interfaces; multiple external interaction interfaces can correspond to multiple computation graph operator interfaces. Computational quantity statistics can be related data on the computational quantity of the corresponding two-tiered API. Computational quantity statistics can include, but are not limited to, input shape, data type, output shape, and parameter names corresponding to the input and output shapes. Input shape refers to the input tensor dimension structure. Output shape refers to the output tensor dimension structure.

[0039] In this embodiment of the invention, the two-level application programming interfaces (APIs) of the modules in the set of modules to be analyzed can be determined first, thereby obtaining the relevant configuration parameters of the two-level APIs of the modules in the set of modules to be analyzed, and obtaining the computational quantity statistical correlation data, that is, obtaining the computational quantity statistical correlation data of the external interaction interface and the computation graph operator interface of the modules in the set of modules to be analyzed.

[0040] Step 130: Calculate the computational workload of the external interaction interface and the computational workload of the computation graph operator interface based on the statistical correlation data.

[0041] The computational complexity of the external interaction interface can be the computational complexity of the external interaction interface itself. Similarly, the computational complexity of the computation graph operator interface can be the computational complexity of the computation graph operator interface.

[0042] In this embodiment of the invention, the computational quantity of the external interaction interface can be calculated based on the computational quantity statistical correlation data of the external interaction interface of the module in the set of modules to be analyzed, and the computational quantity of the computational graph operator interface can be determined based on the computational quantity statistical correlation data of the computational graph operator interface of the module in the set of modules to be analyzed.

[0043] Step 140: Automatically evaluate model anomalies based on the computational load of the external interaction interface and the computational load of the computation graph operator interface.

[0044] In this embodiment of the invention, the difference between the computational amount of the external interaction interface and the computational amount of the computation graph operator interface can be used to determine whether the model framework connection is abnormal, so as to complete the automatic evaluation of model abnormality. It can also compare the computational amount of the computation graph operator interface of the corresponding computation graph of the model before and after the deep learning framework optimization to determine whether the model optimization is effective, that is, to complete the automatic evaluation of the model optimization result.

[0045] The technical solution of this invention obtains the neural network model to be evaluated constructed by a deep learning framework, divides the computation graph of the neural network model to be evaluated into a set of modules to be analyzed, thereby determining the computational quantity statistical correlation data of the two-level application interface of the set of modules to be analyzed. Then, based on the computational quantity statistical correlation data, the computational quantity of the external interaction interface and the computational quantity of the computation graph operator interface are calculated. Furthermore, based on the computational quantity of the external interaction interface and the computational quantity of the computation graph operator interface, model anomaly evaluation is automatically performed. This can realize automatic evaluation of model computational quantity, and by comparing the computational quantity of the two-level interface, framework docking anomalies can be quickly located, thereby improving the development efficiency and running performance of deep learning models. It solves the problems of high manual cost and low model evaluation efficiency in model computational quantity evaluation and framework docking anomaly location, and can break the dependence on manual labor in model computational quantity evaluation and framework docking anomaly location, greatly improving the identification efficiency of abnormal models.

[0046] Example 2

[0047] Figure 2 This is a flowchart of a model anomaly evaluation method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and is further specified, providing specific optional implementation methods for calculating the computational workload of external interaction interfaces and the computational workload of graph operator interfaces based on statistical correlation data of computational workload. Figure 2 As shown, the method includes:

[0048] Step 210: Obtain the neural network model to be evaluated constructed by the deep learning framework, and divide the computation graph of the neural network model to be evaluated into a set of modules to be analyzed.

[0049] Step 220: Determine the computational complexity statistics of the two-level application programming interfaces of the module set to be analyzed.

[0050] Step 230: Determine the first calculation formula based on the external interaction interface input shape, data type, and name of the external interaction interface in the computational quantity statistics related data.

[0051] Here, the input shape of the external interaction interface can be understood as the input shape of the external interaction interface. The name of the external interaction interface can be used to distinguish the external interaction interface corresponding to different modules in the set of modules to be analyzed. The first computational quantity formula can be a formula for calculating the computational quantity of the external interaction interface.

[0052] In this embodiment of the invention, a module in the set of modules to be analyzed that matches the name of the external interaction interface can be determined. Then, the input shape of the external interaction interface corresponding to the module and the data type of the external interaction interface can be parsed from the computational quantity statistical correlation data. Then, combined with the semantics of the matching of the external interaction interface name, the first computational quantity formula can be determined.

[0053] Step 240: Determine the second computational quantity formula based on the input shape, data type, and name of the computational graph operator interface in the computational quantity statistics associated data.

[0054] Here, the input shape of the computation graph operator interface can be understood as the input shape of the computation graph operator interface. The second computational quantity formula can be a formula for calculating the computational quantity of the computation graph operator interface. The name of the computation graph operator interface can be used to distinguish the computation graph operator interface corresponding to different modules in the set of modules to be analyzed.

[0055] In this embodiment of the invention, a module in the set of modules to be analyzed that matches the name of the computation graph operator interface can be identified. Then, the input shape of the computation graph operator interface corresponding to the module and the data type of the computation graph operator interface can be parsed from the computational quantity statistical correlation data. Then, combined with the computational relationship corresponding to the computation graph operator interface, the second computational quantity formula can be determined.

[0056] Step 250: Calculate the computational quantity of the external interaction interface according to the first computational quantity formula, and calculate the computational quantity of the computation graph operator interface according to the second computational quantity formula.

[0057] In this embodiment of the invention, the computational quantity of the corresponding external interaction interface is calculated based on the automatically generated first computational quantity formula, and the computational quantity of the corresponding computation graph operator interface is calculated based on the automatically generated second computational quantity formula.

[0058] Step 260: Automatically evaluate model anomalies based on the computational load of the external interaction interface and the computational load of the computation graph operator interface.

[0059] In an optional embodiment of the present invention, automatic model anomaly assessment is performed based on the computational load of the external interaction interface and the computational load of the computation graph operator interface. This may include: calculating the difference between the computational load of the external interaction interface and the computational load of the computation graph operator interface; when the difference between the computational loads of the two layers is greater than the tolerance threshold, the model anomaly is determined to be caused by an anomaly in the operator docking of the deep learning framework.

[0060] The difference in computational complexity between the two layers can be the difference between the computational complexity of the external interaction interface and the computational complexity of the computation graph operator interface for the same module in the set of modules to be analyzed. The tolerance threshold can be a pre-determined maximum allowable difference between the computational complexity of the external interaction interface and the computation graph operator interface for the same module during normal operator docking of a deep learning framework.

[0061] In this embodiment of the invention, the computational quantity of the external interaction interface matching the same module in the set of modules to be analyzed can be calculated, and the computational quantity difference between the external interaction interface and the computational quantity of the computation graph operator interface can be obtained to obtain the two-layer computational quantity difference value. Then, the two-layer computational quantity difference value is compared with the tolerance threshold. If the two-layer computational quantity difference value is greater than the tolerance threshold, it is determined that the model abnormality is due to the deep learning framework having an abnormal operator docking.

[0062] In an optional embodiment of the present invention, after determining that the cause of the model anomaly is an anomaly in the operator docking of the deep learning framework, the method may further include: obtaining the first anomaly model of operator docking; and debugging the first anomaly model of operator docking.

[0063] Among them, the first operator docking anomaly model can be the module that is the first to experience an anomaly in the set of modules to be analyzed.

[0064] In this embodiment of the invention, the module that first appears abnormal in the set of modules to be analyzed can be identified, namely the first operator docking abnormal model. Then, debugging and analysis can be performed on the first operator docking abnormal model, which can be debugged and analyzed from the source of the abnormality, greatly reducing the workload of debugging and analysis.

[0065] In an optional embodiment of the present invention, after calculating the computational load of the external interaction interface and the computational load of the computational graph operator interface based on the computational load statistical correlation data, the method may further include: obtaining a reference value for the computational load of the computational graph operator interface of the initial neural network model; and evaluating model optimization anomalies based on the reference value for the computational load of the computational graph operator interface and the computational load of the computational graph operator interface.

[0066] The initial neural network model can be a neural network model built before optimization by the deep learning framework, and can be understood as one of the neural network models to be evaluated. The reference value for the computational complexity of the computational graph operator interface can be the computational complexity of the computational graph operator interface of the initial neural network model.

[0067] In this embodiment of the invention, the initial neural network model created before neural network model optimization can be obtained, and then the computational quantity of the computation graph operator interface of the initial neural network model can be determined to obtain a reference value of the computational quantity of the computation graph operator interface. Then, the reference value of the computational quantity of the computation graph operator interface and the computational quantity of the computation graph operator interface are compared. If the difference between the computational quantity of the computation graph operator interface and the reference value of the computational quantity of the computation graph operator interface meets the expectation, that is, the model optimization target is met, then the model optimization is determined to be normal. If the model optimization target is not met, then the model optimization is determined to be abnormal.

[0068] In an optional embodiment of the present invention, evaluating model optimization anomalies based on the reference value of computational computation of the computational graph operator interface and the computational computation of the computational graph operator interface may include: calculating the reduction ratio of model optimization computational computation based on the reference value of computational computation of the computational graph operator interface and the computational computation of the computational graph operator interface; and determining model optimization anomalies when the reduction ratio of model optimization computational computation is less than a preset ratio threshold.

[0069] The percentage decrease in computational cost during model optimization can be used to measure the reduction in computational cost of the computational graph operator interface for creating models before and after optimization of the deep learning framework. The preset percentage threshold can be a pre-set threshold for determining whether model optimization is effective, i.e., the minimum percentage decrease in computational cost that model optimization needs to meet.

[0070] In this embodiment of the invention, the difference between the computational quantity reference value of the computational graph operator interface and the computational quantity of the computational graph operator interface can be determined, and the ratio of this difference to the computational quantity reference value of the computational graph operator interface can be calculated to obtain the model optimization computational quantity reduction ratio. This allows it to be determined whether the model optimization computational quantity reduction ratio is less than a preset ratio threshold. If the model optimization computational quantity reduction ratio is less than the preset ratio threshold, it indicates that the model optimization is abnormal and the optimization effect has not been achieved.

[0071] In an optional embodiment of the present invention, the model anomaly evaluation method may further include: performing data type conversion anomaly detection when the computational load of the computation graph operator interface increases exponentially.

[0072] Among them, data type conversion anomaly detection can be used to check the data type operations corresponding to the computation graph operator interface.

[0073] In this embodiment of the invention, if it is determined that the computational workload of the computation graph operator interface has increased exponentially, it is determined that there is a sudden increase in the computational workload of the computation layer, which may indicate an abnormal data type conversion operation. Therefore, the abnormal data type conversion is automatically detected to avoid wasting a lot of time on useless troubleshooting due to ignoring the verification of data types. In practical applications, this can achieve twice the result with half the effort.

[0074] In a specific example, the model anomaly assessment method includes the following main steps:

[0075] 1) Data capture: During the construction and optimization of the computation graph of the neural network model to be evaluated, the computation graph can be divided into multiple modules according to semantics, and the mapping relationship between Python API and computation graph operators, input and output information, and data types in each module can be recorded.

[0076] The construction of neural network models is typically carried out by deep learning frameworks. These frameworks call underlying operator interfaces through Python layer APIs. The construction of the Python layer results in a corresponding computation graph. This computation graph may be modified through optimization, leading to inconsistencies between the called Python APIs and the operator interfaces. Logically, this can be divided into two approaches while maintaining semantic consistency:

[0077] The Python API corresponds to one or more computation graph operator interfaces in the computation graph. Multiple Python APIs may correspond to multiple computation graph operator interfaces. This is because computation graph optimization methods, such as operator fusion, can lead to a situation where a one-to-one correspondence is not possible, and a many-to-many relationship may occur. Based on logical semantics and input / output shapes, the entire computation graph is divided into S modules, resulting in a set of modules to be analyzed. Computational statistics and correlation data can be stored in the following table:

[0078] Table 1. Data storage table for statistical correlation data of computational workload.

[0079]

[0080] In this context, a 1 in the Python API column corresponding to module 1 indicates that module 1 has one Python API. A 1 in the computational graph operator interface column corresponding to module 1 indicates that module 1 has one computational graph operator interface. A 1 in the Python API column corresponding to module x indicates that module x has one Python API. 1...N in the computational graph operator interface column corresponding to module x indicates that module x has N computational graph operator interfaces. 1...M in the Python API column corresponding to module s indicates that module s has M Python APIs. 1...N in the computational graph operator interface column corresponding to module s indicates that module s has N computational graph operator interfaces. The data in the second row of the Python API corresponding to module s specifically includes the input, data type, parameter names in the shape, and computational cost. Since the data types are consistent with the data types in the second row of the Python API of the aforementioned modules, they will not be described again.

[0081] 2) Based on the operator semantic mapping table, automatically generate the computational formulas for the Python API layer and the computation graph operator interface; by comparing the computational quantities of the two layers, monitor the graph optimization effect and framework integration anomalies.

[0082] Create an operator semantic mapping table for the computation graph operator interface. For example, the torch.mm operator represents multiplication, and torch.add represents addition, etc. The operator semantic mapping table is shown below:

[0083] Table 2 Operator Semantic Mapping Table

[0084]

[0085] Automatic generation of computational complexity formulas: By using the input shape and data type of the Python API, combined with the corresponding semantics (which correspond to the Python API name), the computational complexity of the corresponding module can be obtained. For example, a:(b,s,h), b:(h,h)c(b,s,h)=a*b; Python API: torch.matmul takes a and b as input, thus deriving the computational complexity of the corresponding module c=2bsh 2 Similarly, the computational cost of the corresponding computation graph operator interface can be obtained. In the example above, b, s, and h in the shape of a:(b,s,h) are given by the model structure parameters, so when saving the input and output shapes, the parameter names (b,s,h) in the corresponding shapes are also saved.

[0086] The logic of the computational formula is as follows: Parse input and output (extract the input and output shape, data type, and parameter names in the shape from the Python API / computation graph operator interface); match computation rules: obtain the computation rules from the mapping table according to the operator type; generate symbolic expressions: substitute the parameter names in the shape into the computation rules to generate the computational formula.

[0087] The computational complexity formula (i.e., the second computational complexity formula) of the computational graph operator interface is created in the same logic as the computational complexity formula (i.e., the first computational complexity formula) of the Python API, and will not be elaborated here.

[0088] During graph and operator optimization of the model, the computational load of the computation graph operator interface for each different module can be evaluated and monitored. The specific process is as follows: Record the reference value of the computational load of the computation graph operator interface before optimization; regenerate the computational load and calculation formula of the optimized operator layer after optimization; if the reduction ratio of the model's optimized computational load is less than the preset ratio threshold (e.g., 10%), it is determined that the optimization has not taken effect; update the reference computational load and reference calculation formula according to the latest calculation results for monitoring.

[0089] The call from the model level to the operator level is coordinated by an intermediate framework. By comparing the computational cost of the Python API (i.e., the computational cost of the external interaction interface, based on the computational cost generated by the user code) and the computational cost of the computation graph operator interface (based on the computational cost generated by the actually executed operators), anomalies that occur in the framework during the operator docking process can be identified. It can also reflect optimization failures caused by operators not being docked after optimization. The specific process is as follows: if the difference in computational cost between the two layers is greater than the tolerance threshold (e.g., 5%), then a docking anomaly is determined to exist.

[0090] For the overall process of the above model anomaly assessment method, please refer to [link / reference]. Figure 3 .

[0091] This solution evaluates the model's computing power based on an automatically generated computational load assessment formula. It monitors for anomalies in framework operations by using computational load indicators under stable conditions. The multi-level design allows for rapid identification of framework issues and related integration vulnerabilities.

[0092] The technical solution of this invention obtains a neural network model to be evaluated constructed by a deep learning framework, divides the computation graph of the neural network model to be evaluated into a set of modules to be analyzed, and then determines the computational quantity statistical correlation data of the two-level application interface of the set of modules to be analyzed. Based on the external interaction interface input shape, data type, and name of the external interaction interface in the computational quantity statistical correlation data, a first computational quantity formula is determined. Further, based on the computation graph operator interface input shape, data type, and name of the computation graph operator interface in the computational quantity statistical correlation data, a second computational quantity formula is determined. Thus, based on the computational quantity of the external interaction interface and the computational quantity of the computation graph operator interface, model anomaly evaluation is automatically performed. This enables automatic evaluation of model computational quantity and, by comparing the computational quantity of the two-level interfaces, quickly locates framework docking anomalies, thereby improving the development efficiency and running performance of deep learning models. It solves the problems of high manual costs and low model evaluation efficiency in model computational quantity evaluation and framework docking anomaly location, breaks the dependence on manual labor in model computational quantity evaluation and framework docking anomaly location, and greatly improves the identification efficiency of abnormal models.

[0093] Example 3

[0094] Figure 4 This is a schematic diagram of a model anomaly evaluation device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes:

[0095] The computation graph partitioning module 310 is used to obtain the neural network model to be evaluated constructed by the deep learning framework and partition the computation graph of the neural network model to be evaluated into a set of modules to be analyzed.

[0096] The computational quantity statistics correlation data determination module 320 is used to determine the computational quantity statistics correlation data of the two-level application programming interfaces of the set of modules to be analyzed; wherein, the two-level application programming interfaces include external interaction interfaces and computation graph operator interfaces.

[0097] The computational load determination module 330 is used to calculate the computational load of external interaction interfaces and computational load of computation graph operator interfaces based on statistical correlation data of computational load.

[0098] The model anomaly assessment module 340 is used to automatically assess model anomalies based on the computational load of the external interaction interface and the computational load of the computation graph operator interface.

[0099] The technical solution of this invention obtains the neural network model to be evaluated constructed by a deep learning framework, divides the computation graph of the neural network model to be evaluated into a set of modules to be analyzed, thereby determining the computational quantity statistical correlation data of the two-level application interface of the set of modules to be analyzed. Then, based on the computational quantity statistical correlation data, the computational quantity of the external interaction interface and the computational quantity of the computation graph operator interface are calculated. Furthermore, based on the computational quantity of the external interaction interface and the computational quantity of the computation graph operator interface, model anomaly evaluation is automatically performed. This can realize automatic evaluation of model computational quantity, and by comparing the computational quantity of the two-level interface, framework docking anomalies can be quickly located, thereby improving the development efficiency and running performance of deep learning models. It solves the problems of high manual cost and low model evaluation efficiency in model computational quantity evaluation and framework docking anomaly location, and can break the dependence on manual labor in model computational quantity evaluation and framework docking anomaly location, greatly improving the identification efficiency of abnormal models.

[0100] Optionally, the computational load determination module 330 is used to determine a first computational load formula based on the external interaction interface input shape, data type, and name of the external interaction interface in the computational load statistical correlation data; determine a second computational load formula based on the computation graph operator interface input shape, data type, and name of the computation graph operator interface in the computational load statistical correlation data; calculate the computational load of the external interaction interface based on the first computational load formula, and calculate the computational load of the computation graph operator interface based on the second computational load formula.

[0101] Optionally, the model anomaly evaluation module 340 is used to calculate the difference between the computational amount of the external interaction interface and the computational amount of the computation graph operator interface; when the difference between the computational amounts of the two layers is greater than the tolerance threshold, the model anomaly is determined to be due to an anomaly in the operator docking of the deep learning framework.

[0102] Optionally, the model anomaly evaluation device further includes a debugging module for obtaining the first operator docking anomaly model and debugging the first operator docking anomaly model.

[0103] Optionally, the model anomaly assessment device further includes a model optimization assessment module, used to obtain a reference value for the computational complexity of the computational graph operator interface of the initial neural network model; and to assess model optimization anomalies based on the reference value and the computational complexity of the computational graph operator interface.

[0104] Optionally, the model optimization evaluation module is used to calculate the reduction ratio of model optimization computational load based on the reference value of computational load of the computation graph operator interface and the computational load of the computation graph operator interface; and to determine that the model optimization is abnormal when the reduction ratio of model optimization computational load is less than a preset ratio threshold.

[0105] Optionally, the model anomaly evaluation device also includes a data type conversion anomaly detection module, which is used to detect data type conversion anomalies when the computational load of the computation graph operator interface increases exponentially.

[0106] The model anomaly assessment device provided in this embodiment of the invention can execute the model anomaly assessment method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0107] Example 4

[0108] Figure 5 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0109] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as ROM 12 or RAM 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An I / O interface 15 is also connected to the bus 14. The ROM 12 is a read-only memory, the RAM 13 is a random access memory, and the I / O interface 15 is an input / output interface.

[0110] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0111] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as model anomaly evaluation methods.

[0112] In some embodiments, the model anomaly evaluation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the model anomaly evaluation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the model anomaly evaluation method by any other suitable means (e.g., by means of firmware).

[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0115] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0117] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0118] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS servers, such as high management difficulty and weak business scalability.

[0119] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the model anomaly evaluation method provided in any embodiment of this application. This program product shares the same inventive concept as the model anomaly evaluation methods disclosed in the embodiments of this application, and therefore will not be described in detail here.

[0120] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for evaluating model anomalies, characterized in that, include: Obtain the neural network model to be evaluated constructed by the deep learning framework, and divide the computation graph of the neural network model to be evaluated into a set of modules to be analyzed; Determine the computational complexity statistics of the two-level application programming interfaces (APIs) of the set of modules to be analyzed; wherein the two-level APIs include external interaction interfaces and computation graph operator interfaces; Based on the statistical correlation data of the computational workload, calculate the computational workload of the external interaction interface and the computational workload of the computation graph operator interface; Based on the computational load of the external interaction interface and the computational load of the computation graph operator interface, the model anomaly assessment is performed automatically.

2. The method according to claim 1, characterized in that, Based on the aforementioned computational quantity statistical correlation data, the computational quantity of the external interaction interface and the computational quantity of the computation graph operator interface are calculated, including: Based on the external interaction interface input shape, data type, and name of the external interaction interface in the computational quantity statistical correlation data, determine the first computational quantity formula; The second computational quantity formula is determined based on the input shape, data type, and name of the computational graph operator interface in the computational quantity statistical correlation data. The computational quantity of the external interaction interface is calculated according to the first computational quantity formula, and the computational quantity of the computation graph operator interface is calculated according to the second computational quantity formula.

3. The method according to claim 1, characterized in that, Based on the computational load of the external interaction interface and the computational load of the computation graph operator interface, automatic model anomaly evaluation is performed, including: Calculate the difference between the computational cost of the external interaction interface and the computational cost of the computation graph operator interface; When the difference in computational complexity between the two layers exceeds the tolerance threshold, the cause of the model anomaly is determined to be an anomaly in the operator docking of the deep learning framework.

4. The method according to claim 3, characterized in that, After determining that the model anomaly was caused by an issue with operator docking within the deep learning framework, the following steps were also taken: Obtain the initial operator docking anomaly model; The initial operator docking anomaly model was debugged.

5. The method according to claim 1, characterized in that, After calculating the computational workload of the external interaction interface and the computational workload of the computation graph operator interface based on the statistical correlation data, the method further includes: Obtain reference values ​​for the computational complexity of the computation graph operator interface of the initial neural network model; The model optimization anomalies are evaluated based on the reference value of the computational computation of the computation graph operator interface and the computational computation of the computation graph operator interface.

6. The method according to claim 5, characterized in that, Based on the reference value of the computational cost of the computation graph operator interface and the computational cost of the computation graph operator interface, an evaluation of model optimization anomalies is performed, including: Based on the reference value of the computational workload of the computation graph operator interface and the computational workload of the computation graph operator interface, the computational workload of the optimized computational model is reduced by a certain percentage. When the rate of decrease in the computational load of model optimization is less than a preset threshold, the model optimization is determined to be abnormal.

7. The method according to claim 1, characterized in that, Also includes: When the computational load of the computation graph operator interface increases exponentially, data type conversion anomaly detection is performed.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the model anomaly evaluation method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the model anomaly evaluation method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the model anomaly evaluation method according to any one of claims 1-7.

Images