Image classification method based on cross-frame model conversion and operator difference alignment

Abstract

The invention discloses an image classification method based on cross-frame model conversion and operator difference alignment, which comprises the following steps of: comparing and analyzing operator lists of a source deep learning frame and a target deep learning frame, and mapping to obtain an operator mapping table SS; converting the source framework image classification model into a source AST, and selecting a framework related node set NN; analyzing the context information CC of the node in the NN in the source AST; traversing the source AST, performing cross-frame mapping, aligning operator differences of each node in the NN in cross-deep learning frame mapping by combining CC and SS to obtain a frame related node set NN'after cross-frame mapping, and replacing the NN in the source AST with the NN 'to obtain a target frame image classification model; loading a source deep learning framework official model weight for the target framework image classification model to obtain an image classification reasoning model; and the image classification inference model classifies the to-be-classified images and outputs a classification result. The method can improve the image classification precision and compatibility, and remarkably reduces the time.

Description

Technical Field

[0001] The present invention relates to a method for image classification using an intelligent model in the field of computer vision, and in particular to an image classification method based on cross-frame model conversion and operator difference alignment. Background Art

[0002] Image classification is a core task in the field of computer vision, and it plays a key role in many modern technology applications, such as self-driving vehicles, medical diagnosis, security monitoring, and content filtering. Accurate image classification can not only improve the efficiency of these systems, but also greatly improve their safety and reliability. As technology develops, image classification methods are also evolving, from traditional machine learning techniques to complex deep learning models, each of which aims to improve classification accuracy and handle more complex image data.

[0003] Existing image classification methods can be mainly divided into those based on traditional machine learning and those based on deep learning. In the field of traditional machine learning, common methods include support vector machines (SVM), which distinguish different categories by finding an optimal hyperplane; decision trees and random forests, which perform classification by building tree models of decision rules; and K-nearest neighbors (K-NN), which determines the category of a new sample based on the categories of the nearest few training samples. Among deep learning methods, convolutional neural networks (CNNs) are widely used in automatic feature extraction and image recognition tasks due to their powerful image processing capabilities. Variants such as deep belief networks (DBNs) and long short-term memory networks (LSTMs) are also used to process complex image sequence data. In addition, generative adversarial networks (GANs) generate and improve images by training generators and discriminators, thereby enhancing the performance of classification models. In addition to these basic methods, there are also classification techniques specifically targeting specific functions, such as fine-grained image classification, multi-label classification, and scene recognition, which enable classification tasks to be performed at a more detailed or broader level to meet different application requirements. The selection and application of these methods depend on the specific task requirements and data characteristics. Among them, deep learning technology has become the preferred method for image classification due to its excellent performance.

[0004] Although existing image classification methods have demonstrated high accuracy and excellent performance in many applications, there are still major challenges in the construction and training of image classification models. Building and training accurate image classification models is very time-consuming and complex. Training models requires a lot of mathematical calculations and parameter adjustments, especially when large data sets are involved. The training process may take weeks or even months. However, existing mature and verified image classification models have demonstrated their efficient performance. In this context, cross-framework model conversion technology can bring important conveniences to image classification tasks, especially for those image classification models that have been mature and verified in other frameworks. Cross-framework model conversion technology can be used to achieve rapid reuse of image classification models, avoiding retraining and building models from scratch. This approach can greatly reduce the consumption of time and resources, and ensure that the model can still maintain high accuracy and efficiency in different platforms and environments.

[0005] Existing image classification methods based on cross-frame model conversion mainly include model conversion image classification methods based on computational graphs and APIs. The computational graph represents the computational process of the deep learning model as a directed acyclic graph (DAG) and maps the graph structure between different frameworks. In this method, each node in the graph represents an operator, and the edge represents the dependency relationship between operators. In this way, the computational graph can clearly represent the various hierarchical structures of the image classification model, such as the computational process of operators such as convolutional layers, pooling layers, and fully connected layers. In the image classification task, the computational graph is used for feature extraction and classification decisions. By calculating the nodes in the graph and propagating information along the dependency relationship, the image classification result is finally achieved. For example, Bai X et al. published "ONNX: Open Neural Network Exchange for Framework Interoperability" at NeurIPS19 and Chen T et al. published "TVM: End-to-End Optimization Stack for Deep Learning" at OSDI 18. The APIs-based model conversion image classification methods use the standardized API interfaces provided by the deep learning framework for representation and cross-frame mapping. In the image classification task, the standardized API interfaces can be used to build various layers of the model, such as convolutional layers, pooling layers, activation layers, etc. Each API interface corresponds to a specific operation, and users can define a complete image classification model by combining these interfaces. For example: Baidu's PaConvert, Huawei's TensorlayerX. Interestingly, although these image classification methods based on cross-frame model conversion provide significant convenience and can efficiently ensure the compatibility and classification accuracy of image classification models, they still have certain limitations. These methods fail to effectively solve the operator differences in the cross-frame process (that is, the usage (syntactic differences) and functions (semantic differences) of functional operators between frameworks may be inconsistent), especially in image classification tasks, these differences potentially lead to runtime errors (compatibility) and accuracy degradation (performance) of image classification models.

[0006] Operator syntax defines the operator calling method and name, which is reflected in the operator interface and the way of using the operator. In image classification tasks, the differences in operator syntax are relatively simple and generally only have a potential impact on the compatibility of image classification models. For example, the momentum parameter of the BatchNorm operator of TensorFlow and PyTorch has different names and usages, which may cause compatibility issues such as runtime errors or gradient explosions in the image classification model; operator semantics defines the specific behavior and calculation logic of the operator, which is rooted in the source code implementation in the deep learning framework. In image classification tasks, differences in operator semantics can potentially affect the output and classification results of the model. For example, in PyTorch, the MaxPool2d operator uses dilationW and dilationH parameters, which affect the size and shape of the pooling window, while PaddlePaddle does not support this function. Such differences may cause the image classification model to fail to run correctly after conversion, affecting the final classification accuracy. In order to ensure the compatibility and classification accuracy of image classification models, it is necessary to develop image classification methods for cross-framework model conversion that align operator differences.

[0007] In summary, the main challenge faced by image classification methods based on cross-framework model conversion is the operator differences between functional operators of different frameworks. These differences may cause runtime errors or reduced classification accuracy in image classification models. How to align operator differences to improve the compatibility and performance of converted intelligent models has become a key issue in improving technology in this field. Summary of the invention

[0008] The technical problem to be solved by the present invention is that the construction and training of image classification models by existing deep learning-based image classification methods is very time-consuming and complex, and the existing image classification methods based on cross-frame model conversion fail to effectively solve the problems of operator differences in the cross-frame process. An image classification method based on cross-frame model conversion and operator difference alignment is provided, aiming to improve the classification accuracy of the image classification method and the compatibility of the image classification model, reduce errors reported during the runtime of the image classification model, and improve the image classification accuracy.

[0009] The technical solution of the present invention is: construct an image classification system composed of an operator difference analysis module, an operator difference mapping module, a model syntax parsing module, an operator analysis module, a cross-frame mapping module, an image classification model weight loading module, an image classification module, etc.; select a source frame and a target frame, analyze the operator difference and generate a mapping table; through the model syntax parsing module, the operator analysis module and the cross-frame mapping module, the source learning frame image classification model uploaded by the user is converted, and the operator difference in the source deep learning frame image classification model is aligned in combination with the operator difference mapping table and the cross-frame mapping is completed to obtain the target frame image classification model. Then, the model weight conversion format officially provided by the source deep learning framework is loaded into the target frame image classification model to obtain an image classification module composed of an image classification inference model. Finally, the image classification module identifies the image uploaded by the user, performs the classification task and outputs the classification result. The present invention converts and aligns the operator difference between the source deep learning frame image classification model and the target deep learning frame image classification model during the conversion process through the cross-frame model, and does not need to build and train the image classification model, which greatly saves time, improves the compatibility of the image classification model, and improves the image classification accuracy.

[0010] The present invention comprises the following steps:

[0011] The first step is to build an image classification system based on cross-frame model conversion and operator difference alignment. The system consists of an operator difference analysis module, an operator difference mapping module, a model syntax parsing module, an operator analysis module, a cross-frame mapping module, an image classification model weight loading module, and an image classification module.

[0012] The operator difference analysis module is an operator difference extraction tool, which is connected to the operator difference mapping module. It compares and analyzes the source deep learning framework operator list SO and the target deep learning framework operator list TO input by the user, extracts the differences in the operator interface and operator function code of the operators corresponding to SO and TO, and obtains the operator name, operator interface mapping relationship, operator difference type, operator function code comparison fragment and operator function call sequence. The operator name is a string type data, which stores the source deep learning framework operator name. The operator interface mapping relationship is a dictionary type data, including two key-value pairs (a key-value pair is a tuple consisting of a key and a value), namely the operator name mapping relationship key-value pair (source deep learning framework operator name, target deep learning framework operator name), parameter mapping relationship key-value pair (key storage source deep learning framework operator parameters, target deep learning framework operator parameters); the operator difference type is a string type data, including three types, namely "operator function signature and usage differences", "operator behavior differences" and "operator missing differences"; the operator function code comparison fragment is a hierarchical dictionary, the root level contains a key-value pair (the associated source deep learning framework parameters in the function code, the modification sequence). The modification sequence (array type) contains multiple modification operations, each modification operation describes the specific modification made to the source framework operator function code, and each modification operation is a sub-dictionary containing the following three fields: programming language type (string type, such as: Python, C), function code fragment context information (dictionary type), operation list (containing multiple specific modification operations, each operation is a dictionary type). The function code snippet context information contains three key-value pairs: sub-function call key-value pair ("sub-function call", sub-function call list), header file key-value pair ("header file", header file code), namespace key-value pair ("namespace", the namespace of the source framework operator function code). The operation list contains multiple operations, each of which contains two key-value pairs: modification position key-value pair ("modification position", the line number of the code in the function where the function code is located), modification code key-value pair (source framework function code snippet, target framework function code snippet). Each element in the operator function call sequence is an operation, corresponding to one of the target framework operator, source framework operator parameter, and basic operator. The operator name, operator difference type, operator interface mapping relationship, operator function code comparison snippet, and operator function call sequence are sent to the operator difference mapping module.

[0013] The operator difference mapping module is connected to the operator difference analysis module, the model syntax parsing module, and the cross-frame mapping module, and receives the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment, and operator function call sequence from the operator difference analysis module. Map the above content into the operator mapping table SS. SS is a hierarchical dictionary with a root level of a key-value pair list. In each key-value pair in the key-value pair list, the key is the operator name and the value is a dictionary. It contains two or three key-value pairs according to the operator difference type: when the operator difference type is "operator function signature and usage difference", the value contains the operator difference type and operator interface mapping relationship key-value pair ("operator interface mapping relationship", operator interface mapping relationship); when the operator difference type is "operator behavior difference", the value contains the operator difference type, operator interface mapping relationship key-value pair, operator function code comparison fragment key-value pair ("operator function code comparison fragment", storage operator function code comparison fragment); when the operator difference type is "operator missing difference", the value contains the operator difference type key-value pair, operator interface mapping relationship key-value pair, operator function call sequence key-value pair ("operator function call sequence", operator function call sequence). Send the operator mapping table SS to the model syntax parsing module and the cross-framework mapping module.

[0014] The model syntax parsing module is connected to the operator difference mapping module, the operator analysis module, the cross-frame mapping module, and the image classification model weight loading module. It receives the source frame image classification model uploaded by the user, receives the operator mapping table SS from the operator difference mapping module, obtains the operator name O and the source frame image classification model uploaded by the user from SS, and parses the source frame image classification model uploaded by the user into a source abstract syntax tree (referred to as source AST). An abstract syntax tree is a grammatical structure that expresses a model in a tree structure. In this structure, each node represents a basic component (operator, function, module) in the image classification model, and the relationship between nodes represents the hierarchy and dependency of these components. The source AST represents the various components of the image classification model in an abstract way that is independent of the framework. The model syntax parsing module identifies and marks the nodes related to the operator in the source AST, and filters out the frame-related node set NN from the source AST through the frame root directory name, sends NN and the source AST to the operator analysis module, and sends NN to the cross-frame mapping module.

[0015] The operator analysis module is connected to the model syntax parsing module and the cross-frame mapping module. It receives the frame-related node set NN and the source AST from the model syntax parsing module, analyzes each node in NN, and obtains the context information CC of each node in NN in the source AST. The context information includes the operator name, operator parameter list (each element includes parameter name, parameter value and parameter default value), node type (ast.Call or ast.attribute, ast is a standard library of Python), node position (that is, the code line where the node is located), and pointers to parent nodes and child nodes. CC and the source AST are sent to the cross-frame mapping module.

[0016] The cross-frame mapping module is connected to the operator difference mapping module, the model syntax parsing module, the operator analysis module, and the image classification model weight loading module. It receives the operator mapping table SS from the operator difference mapping module, receives the framework-related node set NN from the model syntax parsing module, receives the operator's context information CC and source AST from the operator analysis module, and receives the target framework CMakeList template uploaded by the user. It performs cross-frame mapping on each node in NN, and performs different difference alignment operations according to the operator difference type of each node in NN corresponding to the operator in SS. The alignment process needs to complete the operator interface mapping (using the operator interface mapping relationship in SS to complete the operator API mapping), the target framework operator Python function code alignment (encapsulate the aligned Python function code into a library function, and import it into the target framework image classification model to replace the source framework operator), and the target framework operator C function code alignment (use the CMakeList template to compile the C function code into a .so file, and link these .so files in the target framework image classification model to achieve C function code alignment). Finally, the mapped frame-related node set NN' is obtained, and according to the node position in the operator context CC, the nodes in NN in the source AST are replaced with the corresponding nodes in NN' to obtain the abstract syntax tree of the target frame (referred to as target AST), and the target AST is reparsed into the target frame image classification model, and the target frame image classification model is sent to the image classification model weight loading module.

[0017] The image classification model weight loading module is connected to the cross-frame mapping module and the image classification module. It receives the official model weights of the source deep learning framework uploaded by the user, receives the target framework image classification model from the cross-frame mapping module, converts the format of the official model weights of the source deep learning framework and loads them into the target framework image classification model to obtain the image classification inference model, and sends the image classification inference model to the image classification module.

[0018] The image classification model module is connected to the image classification model weight loading module, receives the image to be classified uploaded by the user, receives the image classification inference model from the image classification model weight loading module, inputs the image to be classified uploaded by the user into the image classification inference model, and obtains the final classification result of the image to be classified.

[0019] In the second step, the operator difference analysis module compares the operator list SO of the source deep learning framework and the operator list TO of the target deep learning framework, and sends the analysis results, i.e., the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence, to the operator difference mapping module. The method is:

[0020] 2.1 Operator difference analysis module receives the operator list SO of the source deep learning framework and the operator list TO of the target deep learning framework input by the user, SO = {so 1 ,so 2 ,…,so i ,…,so I}, so i is the i-th operator in SO, I is the total number of operators in the source deep learning framework, 1≤i≤I; TO={to 1 ,to 2 ,…,to j ,…,to J}, to j is the jth operator in TO, J is the total number of operators in the target deep learning framework, 1≤j≤J;

[0021] 2.2 Traverse SO, find the matching operator of each operator to be analyzed in SO in TO, and analyze the function codes of the two matching operators to obtain the function code comparison fragments of the two matching operators. The method is:

[0022] 2.2.1 Construct an empty list NAME_TO and put to 1 ,to 2 ,…,to j ,…,to J Place it in NAME_TO.

[0023] 2.2.2 Initialize i=1;

[0024] 2.2.3 For the i-th operator so in SO i , will so iand NAME_TO are input to the get_close_matches function in Python's difflib library. The function of the get_close_matches function is to find several strings that are most similar to the target string in a string list. The get_close_matches function is based on so i The operator name in NAME_TO is searched for an operator name that matches the operator name in NAME_TO, and a list containing all the operator names in NAME_TO that match the operator name in NAME_TO is obtained. i List of approximate matching operator names LL i If LL i Not empty, execute 2.2.3.1; if LL i If it is empty, execute 2.2.3.2;

[0025] 2.2.3.1 Find LL in TO i The first element of finds the corresponding target frame operator in TO, let to j , to j for i Matching operator, go to 2.2.4;

[0026] 2.2.3.2 Explanation: TO does not exist with so i Match the target frame operator, go to 2.2.5.

[0027] 2.2.4 At this time, TO exists with so i The target frame operator to match j , analyze so i and to j The function code is obtained i and to j Function code comparison snippet, the method is:

[0028] 2.2.4.1 Generate so i and to j The interface mapping relationship is as follows:

[0029] 2.2.4.1.1 Record so i and to j Operator name, formed so i and to j Operator name mapping key-value pairs (so i The operator name, to j operator name).

[0030] 2.2.4.1.2 Initialize so i Parameter Mapping List SL i and the failed mapping parameter list fo i Is empty. Traverse soi Operator parameter list param_so i ,param_so i = {param_so i1 ,param_so i2 ,…,param_so ik ,…,param_so iK}, param_so ik param_so i The kth parameter, K is the parameter in param_so i The total number of parameters, 1≤k≤K. Set the operator parameter name param_so ik As a keyword, in to j The parameter list param_to j Search in param_to j = {param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL}, L is param_to j The total number of parameters in ,1≤l≤L. That is, param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL with param_so ik Perform name matching in to j Find the value corresponding to param_so ik The corresponding operator parameters. The specific method is:

[0031] 2.2.4.1.2.1 Initialize k = 1;

[0032] 2.2.4.1.2.2 Traversing to j The parameter list param_to j , param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL with param_so ik To match, use:

[0033] 2.2.4.1.2.2.1 Initialize l = 1;

[0034] 2.2.4.1.2.2.2 If param_soik and param_to jl The names of the same, or their prefixes or suffixes partially match, then param_so ik and param_to jl Matching is successful. If the match is successful, a key-value pair (param_so ik , param_to jl ), go to 2.2.4.1.2.3; if that fails, go to 2.2.4.1.2.2.3;

[0035] 2.2.4.1.2.2.3param_so ik and param_to jl Matching failed, set l = l + 1. If l>L, go to 2.2.4.1.2.4; if l≤L, go to 2.2.4.1.2.2.2;

[0036] 2.2.4.1.2.3 At this time param_so ik and param_to jl Matching is successful, (param_so ik , param_to jl ) Add parameter mapping list SL i , let k=k+1, if k>K, go to 2.2.4.1.2.5; if k≤K, go to 2.2.4.1.2.2;

[0037] 2.2.4.1.2.4 Explanation of param_so at this time ik With the target frame operator to j If all parameters in the match fail, param_so ik Add param_so i Failed mapping parameter list FO i , go to 2.2.4.1.2.5;

[0038] 2.2.4.1.2.5 Now complete so i All parameters match, forming so i Parameter Mapping List SL i and so i Failed to map parameter list fo i , SL i ={sl i1 ,sl i2 ,…,sl it ,…,sl iT}, sl it For SL i The tth so i and toj Parameter mapping key-value pairs, T is so i and to j The total number of parameter mapping key-value pairs; FO i ={fo i1 ,fo i2 ,…,fo im ,…,fo iM}, for im For FO i The mth mapping failure parameter in , M is FO i The total number of all failed parameters in;

[0039] 2.2.4.1.3 Build the operator interface mapping relationship dictionary. The operator interface mapping relationship dictionary contains two key-value pairs, one for the so formed in step 2.2.4.1.1 i and to j The operator name mapping key-value pair and the parameter mapping list key-value pair ("parameter mapping list", SL i ), the dictionary is the operator interface mapping relationship;

[0040] 2.2.4.2 Traversing FO i , follow up i1 ,fo i2 ,…,fo im ,…,fo iM Related operator implementation codes, obtain operator function code comparison snippets (that is, track and record the code lines containing the parameter), the method is:

[0041] 2.2.4.2.1 Initialize m = 1;

[0042] 2.2.4.2.2 Use PDB (Python debugger) and GDB (GNU debugger) to run the operator so i and to j Among them, PDB is used to debug the code modules written in Python, providing functions such as step-by-step execution, breakpoint setting, and variable inspection. im Related Python function code comparison snippets; GDB is used to debug low-level code written in C / C++, supporting memory inspection, thread debugging and performance analysis, and obtaining fo im Related C function code comparison snippets. Through this joint debugging method, comprehensive records and fo im Related code running status. Merge fo im Related Python function code comparison snippets and fo im Compare the relevant C function code snippets to get the complete build fo im The relevant function code comparison snippet is as follows:

[0043] 2.2.4.2.2.1 Build fo im Related Python function code comparison snippets: First, use PDB to debug the operator so i and to j , set breakpoints in the __init__ and forward functions in PDB. Then run so i When the execution reaches the breakpoint, it enters debug mode. In debug mode, PDB uses the "enter function" instruction and the "next line" instruction to track the so i Python function code, execute the code line by line, and record so i fo im Related function code (in code line format). For each line so i fo im Related function code, run to j When the execution reaches the breakpoint, it enters the debug mode. In debug mode, PDB uses the "enter function" instruction and the "next line" instruction to track to j Python function code, execute the code line by line, use Python's ast parse function to parse so i fo im The relevant lines of code and to j The current debugging code line uses the tree isomorphism (AHU) algorithm to determine so by matching the node labels and the order and connection relationship of their child nodes. i fo im The subtree structure of the relevant code lines is the same as to j The similarity of the subtree of the currently debugged code line in the local structure is obtained to obtain the similarity score between the two. i fo im The subtree structure of the relevant code lines is the same as to j If the subtree structure of the currently debugged code line matches (similarity score greater than 0.8), then build so i fo im The Python function code sends snippet key-value pairs (so i Function code lines, to j Function code line); if to j Does not exist and so i fo im Related lines of code similar to the code, also build so i fo im Related Python function code snippet key-value pairs (so i function code lines, empty). Build fo imA list of related Python function code contexts, including the called sub-functions in the same file, the function to which the function code snippet belongs, the namespace and header file. im Related Python function code comparison snippets, fo im The relevant Python function code comparison snippet uses a dictionary data type, which contains three key-value pairs: programming language type key-value pair ("programming language type", "Python"), context information key-value pair ("context information", fo im Related Python function code context list), function code list key-value pairs ("function code key-value pair list", so i fo im Related Python function code comparison snippet key-value pair list).

[0044] 2.2.4.2.2.2 Construction of fo im Related C function code comparison snippet: Using GDB to debug the operator so i and to j , extract the function call stack (actually a list) of the operator program stack in GDB, use the keyword list (impl, functor, kernel) to perform suffix matching (that is, the function name ends with the keyword), filter out the core calculation function (kernel function), and set a breakpoint for the kernel function. In debug mode, run so i When the execution reaches the breakpoint, GDB uses the "enter function" instruction and the "next line" instruction to track the so i C function code, execute the code line by line, and record so i fo im Related function code (in code line format). For each line so i fo im Related function code, run to j When the execution reaches the breakpoint, it enters the debug mode. In debug mode, GDB uses the "enter function" instruction and the "next line" instruction to track to j C function code, using Clang (Clang is a powerful C / C++ / Objective-C compiler front end, which can not only compile code, but also generate a detailed abstract syntax tree (AST)) to parse so i fo im Parameter related code lines and to j The current debugged code line uses the tree isomorphism algorithm to determine the similarity of the two trees in local structure by matching the order and connection relationship of node labels and their child nodes, and obtain the similarity score between the two.i fo im The subtree structure of the relevant code lines is the same as to j If the subtree structure of the currently debugged code line matches (similarity score greater than 0.8), then build so i fo im C function code snippet key-value pairs (so i The C function code line, to j C function code line); if to j Does not exist and so i fo im Related lines of code similar to the code, also build so i fo im Related C function code snippet key-value pairs (so i lines of C function code, empty). Build fo im A list of related C function code contexts, including the called sub-functions in the same file, the function to which the function code snippet belongs, the namespace and header file. im Related C function code comparison snippet (dictionary type), fo im The relevant C function code comparison snippet contains three key-value pairs: programming language type key-value pair ("programming language type", "C"), context information key-value pair ("context information", fo im Related C function code context list), function code list key-value pairs ("function code key-value pair list", so i fo im Related C function code comparison snippet key-value pair list).

[0045] 2.2.4.2.2.3 Merge fo im Related Python function code comparison snippets and fo im Compare the relevant C function code snippets to get the complete build fo im Related functional code comparison snippets, complete build fo im The relevant function code comparison snippet contains two key-value pairs: the associated parameter key-value pair ("associated parameter", fo im parameter name), operation sequence key-value pairs ("operation sequence", fo im Related Python function code comparison snippets and fo im The dictionary list formed by combining the relevant C function code comparison fragments). im Related Python function code comparison snippets and fo im The relevant C function code comparison snippet contains the function code list key-value pairs, which are defined as fo im The operation sequence seq_foim , seq_fo im ={seq_fo im1 ,seq_fo im2 ,…,seq_fo imr ,…,seq_fo imR}, seq_fo imr For im The rth function code comparison fragment key-value pair, R is so i fo im Related Python function code comparison snippet key-value pair list and so i fo im The relevant C function code compares the sum of the fragment key-value pair lists).

[0046] 2.2.4.2.3 Let m = m = +1. If m>M, change fo i1 ,fo i2 ,…,fo im ,…,fo iM The relevant functional code is compared and merged to obtain FO i Related function code comparison fragment (dictionary list type), go to 2.2.6; if m≤M, go to 2.2.4.2.2.1;

[0047] 2.2.5 At this time, TO does not exist with so i Matching target frame operator, analysis so i The kernel function of i The operator function call sequence is:

[0048] 2.2.5.1 Using GDB to debug so i , extract the function call stack (actually a list) of the operator program heap in GDB, use framework-related keywords (impl, functor, kernel) to query so i Perform suffix matching on the function calls in the function call stack (i.e., the function call name ends with a keyword), and filter out so i kernelfunction in the function call stack.

[0049] 2.2.5.2 Use the function syntax analysis method to analyze the core computing function (kernel function) and form the operation sequence of the kernel function according to the actual execution order of the control flow. The method is:

[0050] 2.2.5.2.1 Use Clang to generate an abstract syntax tree for the kernel function and obtain the AST of the kernel function.

[0051] 2.2.5.2.2 Analyze the AST of the kernel function, process the return statement of the kernel function (Clang's returnStmt type will mark the return statement), and record the code line where the return statement is located.

[0052] 2.2.5.2.3 Initialize an empty operation sequence list corresponding to the kernel function, which is used to record all relevant operations in the analysis process (target framework operator, source framework operator parameters, sub-functions called in the same file, arithmetic operators).

[0053] 2.2.5.2.4 Use the syntax expression analysis method to analyze the return statement code of the kernel function and obtain the syntax expression of the return statement. The return statement corresponds to the returnStmt node in the kernel function's AST, which contains a return value expression (the expression is a tree structure). Analyze and identify the local variables (clang's VarDecl node), basic operators, operator parameters (i.e., kernel function parameters, clang's ParmVarDecl node), and function calls (clang's CallExpr node).

[0054] 2.2.5.2.5 Use the depth-first search method to traverse the syntax expression of the return statement in 2.2.5.2.4 to obtain the operation sequence of the kernel function.

[0055] 2.2.5.3 Use the local variable analysis method to analyze all local variables (i.e. VarDecl nodes) in the kernel function's operation sequence. Trace back the assignment process of local variables, obtain the syntax expression of local variables and replace the syntax expression with the VarDecl node in the operation sequence to obtain the kernel function operation sequence that does not contain the VarDecl node. The method is:

[0056] 2.2.5.3.1 Analyze the assignment statement of the local variable (AssignExpr node), the local variable on the left side of the assignment statement, and a syntax expression on the right side of the assignment statement. Use the syntax expression analysis method described in 2.2.5.2.4 to analyze the right side of the assignment statement, analyze the local variables, basic operators, operator parameters, and function calls therein, use the depth-first search method to build the operation sequence of the AssignExpr node syntax expression, and replace the VarDecl node currently being analyzed in the operation sequence of the kernel function.

[0057] 2.2.5.3.2 If there are no local variables (i.e., VarDecl nodes) in the operation sequence of the kernel function, complete the local variable analysis in the return statement to obtain the kernel function operation sequence that does not contain VarDecl nodes, and go to 2.2.5.4; if there are still other VarDecl nodes in the operation sequence of the kernel function, let the local variables currently being analyzed be other VarDecl nodes, and go to 2.2.5.3.1; go to 2.2.5.3.1.

[0058] 2.2.5.4 Use the function call analysis method to further analyze all function calls (i.e., CallExpr nodes) in the kernel function's operation sequence. Each function call operation corresponds to a CallExpr node. Get the name of the function call of the CallExpr node and the parameter list in the function call. Suffix match the function call name with the target framework operator name. If the function call of the CallExpr node matches the target framework operator name successfully, replace the function call of the CallExpr node with the target framework operator; if the function call of the CallExpr node does not match the target framework operator name successfully, then deeply analyze the function call code to form the operation sequence of the CallExpr node, replace the CallExpr node in the kernel function's operation sequence, and get so i The operator function call sequence ordseq_so i The method is:

[0059] 2.2.5.4.1 Get the function name and parameter list of each CallExpr node in the kernel function's operation sequence;

[0060] 2.2.5.4.2 Match the relationship between each CallExpr node in the kernel function's operation sequence and the target framework operator. Check whether the function name in the CallExpr node matches the operator name in the target framework by exact matching or suffix matching (for example, whether the function matrixMultiply matches the operator MatrixMul in the target framework. In addition to matching the function name, it is also necessary to confirm whether the function's parameter type and quantity meet the requirements of the target framework operator). If the function name and parameters match successfully, replace the CallExpr node with the operator of the target framework and go to 2.2.5.4.4; if not, further analyze the CallExpr node and go to 2.2.5.4.3;

[0061] 2.2.5.4.3 Use the function syntax analysis method of 2.2.5.2 to analyze the function call content (i.e., function code) corresponding to the CallExpr node, and form an operation sequence of the CallExpr node according to the actual execution order of the control flow; use the local variable analysis method of 2.2.5.3 to analyze all VarDecl nodes (local variables) in the operation sequence of the CallExpr node, backtrack the assignment process of the local variables, obtain the syntax expression of the local variables and replace the VarDecl node in the operation sequence with the syntax expression to obtain the operation sequence of the CallExpr node that does not contain the VarDecl node; replace the CallExpr node currently being analyzed in the operation sequence of the kernel function with the operation sequence of the CallExpr node that does not contain the VarDecl node.

[0062] 2.2.5.4.4 Continue to traverse the CallExpr nodes in the kernel function's operation sequence. If there is an unanalyzed CallExpr node, go to 2.2.5.4.1; if not, get so i The operator function call sequence ordseq_so i , ordseq_so i ={ordseq_so i1 ,ordseq_so i2 ,…,ordseq_so is ,…,ordseq_so iS}, ordseq_so is is ordseq_so i The sth operation, S is ordseq_so i The total number of operations in the sequence, go to 2.2.6;

[0063] 2.2.6 Let i = i + 1. If i>I, go to 2.3; if i≤I, go to 2.2.3;

[0064] 2.3 will so i The operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment or operator function call sequence are sent to the operator difference mapping module.

[0065] In the third step, the operator difference mapping module receives the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence from the operator difference analysis module, traverses the source deep learning framework operator list SO, and processes SO according to different operator difference types. 1 ,so 2 ,…,so i ,…,so I The operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence are mapped into the operator mapping table SS, and the SS is sent to the model syntax parsing module and the cross-frame mapping module. The method is:

[0066] 3.1 Initialize the operator mapping table SS to be empty, and initialize i=1;

[0067] 3.2 According to so i The operator difference type is handled differently: if the operator difference type is "difference in operator function signature and usage", jump to 3.2.1; if the operator difference type is "difference in operator behavior", jump to 3.2.2; if the operator difference type is "difference in operator missingness", jump to 3.2.3;

[0068] 3.2.1 Construct a hierarchical dictionary to record three key-value pairs: operator name key-value pair ("operator name", so i operator name), operator difference type key-value pair ("operator difference type", so i operator difference type), operator interface mapping key-value pairs ("operator interface mapping", so i operator interface mapping relationship); add the hierarchical dictionary to SS and go to 3.3;

[0069] 3.2.2 Construct a hierarchical dictionary to record four key-value pairs: operator name key-value pair ("operator name", so i operator name), operator difference type key-value pair ("operator difference type", so i operator difference type), operator interface mapping key-value pairs ("operator interface mapping", so i operator interface mapping relationship), operator function code comparison fragment key-value pairs ("operator function code comparison fragment", soi ); Add the hierarchical dictionary to SS and go to 3.3;

[0070] 3.2.3 Build a hierarchical dictionary to record three key-value pairs: operator name key-value pair (key is "operator name", so i operator name), operator difference type key-value pair ("operator difference type", so i operator difference type), operator function call sequence key-value pairs ("operator function call sequence", so i operator function call sequence); add the hierarchical dictionary to SS and go to 3.3;

[0071] 3.3 Let i=i+1. If i>I, it means that SO is processed and SS is obtained, go to 3.4; if i≤I, go to 3.2;

[0072] 3.4 Send SS to the model syntax parsing module and cross-frame mapping module.

[0073] In the fourth step, the model syntax parsing module receives the source frame image classification model uploaded by the user, and receives the operator difference mapping table SS from the operator difference mapping module. The source frame image classification model is converted into AST (referred to as source AST), and the components of the source frame image classification model are represented in a frame-independent manner. The nodes related to the operator are identified and marked, and the frame-related node set NN is filtered out using the root directory name of the frame, and NN is sent to the operator analysis module and the cross-frame mapping module. The method is:

[0074] 4.1 Parse the source framework image classification model into a syntax abstract tree (source AST for short); Use the parse function in the Python standard library ast to convert the source framework image classification model code into AST.

[0075] 4.2 Use the visit function of the source AST to recursively traverse the source AST nodes, identify each Call type or Attribute type node in the source AST, filter out the framework-related nodes, and obtain the related node set NN. The method is:

[0076] 4.2.1 Start traversing from the root node of the source AST, and let the current node v = the root node of the source AST;

[0077] 4.2.2 Determine the node type of v. If the node type of v is a Call type or an Attribute type, go to 4.2.2.2. If the node type of v is neither a Call type nor an Attribute type, go to 4.2.2.1.

[0078] 4.2.2.1 does not mark v and goes to 4.2.2.3;

[0079] 4.2.2.2 Get the operator name in v and traverse so in SS i The operator name of is matched with the operator name in v. If the match is successful, v is determined to be a framework-related node and recorded. The method is:

[0080] 4.2.2.2.1 Initialize the relevant node set NN to be empty, and initialize i=1;

[0081] 4.2.2.2.2 Get so from SS i The operator name of is matched against the operator name of v (obtained by getting the value of v). i If the operator name of and the operator name of v match successfully, the node is marked and v is placed in NN, and the process goes to 4.2.2.3. If the match fails, no marking is performed and the process goes to 4.2.2.2.3.

[0082] 4.2.2.2.3 Let i = i + 1. If i > I, go to 4.2.2.3. If i ≤ I, go to 4.2.2.2.2.

[0083] 4.2.2.3 Traverse the next node of the source AST (call the visit function of AST, which will automatically recursively visit all nodes). If the next node exists, go to 4.2.2; if the next node does not exist, it means that the framework-related node set NN containing all marked nodes is obtained. NN = {n 1 ,n 2 ,…,n p ,…,n P},n p is the p-th frame-related node (operator) of the NN, that is, v, n placed in the NN p It has three attributes: name (operator name), args (operator parameters), and class (node ​​type corresponding to the operator in the source AST). 1≤p≤P, where P is the number of framework-related nodes in the NN. Go to 4.3.

[0084] 4.3 Send NN and source AST to the operator analysis module and cross-frame mapping module.

[0085] In the fifth step, the operator analysis module receives the frame-related node set NN and the source AST from the model syntax parsing module, parses the context information CC of the P frame-related nodes in NN in the source AST, and sends CC to the cross-frame mapping module.

[0086] The method is:

[0087] 5.1 Let p = 1; initialize the context information CC in the source AST to be empty, and initialize the node context information list CC p Is empty.

[0088] 5.2 Get the nth frame-related node n in NN p The context information of the source AST includes: operator name, operator parameter list, node type, node position (the code line where the node is located), and pointers to parent nodes and child nodes. Each parameter in the operator parameter list includes: parameter name, parameter value, and parameter default value. The method is:

[0089] 5.2.1 Use the visit function of the source AST to traverse the source AST and record n p A list of parent nodes and child nodes;

[0090] 5.2.2 Access n p Attribute name, get n p The name of the operator;

[0091] 5.2.3 Access n p The attribute args, get n p The parameter list.

[0092] 5.2.4 Access n p The attribute __class__ of p Node type;

[0093] 5.2.5 Using astor.to_source function to obtain n p The code lines of the node in the source framework image classification model (astor is a standard function library of Python);

[0094] 5.3 replace 5.2 with n p The operator name, operator parameter list, node type, node position, and pointers to parent nodes and child nodes are added to CC p In the p Join CC.

[0095] 5.4 Let p = p + 1. If p > P, go to 5.5. If p ≤ P, go to 5.2.

[0096] 5.5 Send CC to the cross-frame mapping module.

[0097] In the sixth step, the cross-frame mapping module receives SS from the operator difference mapping module, receives NN from the model syntax parsing module, receives CC and source AST from the operator analysis node module, receives the target framework CMakeList template uploaded by the user, traverses the source AST, performs cross-frame mapping, and 1 ,n2 ,…,n p ,…,n P Combine CC and SS to align n 1 ,n 2 ,…,n p ,…,n P The operator differences in the cross-deep learning framework mapping process are used to obtain the framework-related node set NN' after cross-frame mapping, and NN' is used to replace NN in the source AST to obtain the grammatical abstract syntax tree of the target framework (referred to as target AST). The target AST is parsed into the target framework image classification model, and the target framework image classification model is sent to the image classification model weight loading module. The method is:

[0098] 6.1 Let p = 1; initialize the frame-related node set NN' after cross-frame mapping to be empty;

[0099] 6.2 According to n p The operator difference type in the operator difference mapping table SS is used to perform cross-frame mapping and align the operator differences in the cross-frame mapping process to obtain the cross-frame mapping node n. p '. The method is:

[0100] 6.2.1 If the operator difference type is "operator function signature and usage differences", go to 6.2.2; if the operator difference type is "operator behavior differences", go to 6.2.3; if the operator difference type is "operator missing differences", go to 6.2.4;

[0101] 6.2.2 If the operator difference type is "Operator function signature and usage differences", complete n p Interface mapping, get n after cross-frame mapping p '. The method is:

[0102] 6.2.2.1 Reading n in SS p Operator interface mapping relationship, modify n according to the operator name in the mapping relationship p Name, complete n p Operator name mapping;

[0103] 6.2.2.2 Access n p The args attribute (i.e. n p Parameter list), let n p The parameter list is param_n p , param_n p = {param_n p1 ,param_n p2 ,…,param_n po ,…,param_n pO}, param_npo param_n p The oth parameter of p The total number of parameters actually used, 1≤o≤O.

[0104] 6.2.2.3 Access n p __class__. attribute, if the attribute is "attribute", go to 6.3; if the attribute is "Call", go to 6.2.2.4;

[0105] 6.2.2.4 Using the operator parameter mapping method, according to n in SS p Parameter mapping list SL of interface mapping relationship i param_n p After the mapping is completed, the type n p (i.e. the operator type is "difference in operator function signature and usage") can directly get n p ', n p 'Put it to NN'. The method is:

[0106] 6.2.2.4.1 Initialize o = 1;

[0107] 6.2.2.4.2 For the oth param_n po , extract n from SS p Parameter mapping list SL in the interface mapping relationship i , find SL i to param_n po As a key-value pair, param_n po Replace with the corresponding value in the key-value pair;

[0108] 6.2.2.4.3 Let o = o + 1. If o > O, then n is completed. p Interface mapping, obtain node n after cross-frame mapping p ', go to 6.2.2.5; if o≤O, go to 6.2.2.4.2;

[0109] 6.2.2.5 n p 'Put to NN', go to 6.3;

[0110] 6.2.3 For n p Perform operator interface mapping and operator parameter function alignment. The method is:

[0111] 6.2.3.1 Read n p Operator interface mapping relationship, modify n according to the operator name in the mapping relationship p Name, complete n p Operator name mapping;

[0112] 6.2.3.2 Access n p The args attribute (n p .args is n p Parameter list), let n p The parameter list is param_n p .

[0113] 6.2.3.3 Using the operator parameter mapping method of 6.2.2.4, according to n in SS p Parameter mapping list SL of interface mapping relationship i param_n p After the mapping is completed, the type n p (That is, the operator type is "Operator Behavior Difference") Only some parameters are mapped successfully, and some parameters fail to be mapped.

[0114] 6.2.3.4 Collect the failed mapping parameters in 6.2.3.3 and get F_n p , F_n p ={F_n p1 ,F_n p2 ,…,F_n pq ,…,F_n pq}, F_n pq n p The qth failed mapping parameter of , where Q is the source frame operator n p Total number of failed mapping parameters, n p Corresponding to an element in SO, let the element be so i , F_n p So i Failed to map parameter list fo i For F_n p , using n in S p Operator function code comparison snippet, traversing FO i , align F_n p Related function codes, get the node n after cross-frame mapping p '. The method is:

[0115] 6.2.3.4.1 Initialize q = 1;

[0116] 6.2.3.4.2 Find FO i Chinese and Japanese F_n pq The key-value pair as the key. If it exists, the value of the key-value pair is F_n pq For the relevant function code comparison snippet, go to 6.2.3.4.3; if it does not exist, go to 6.2.3.4.4;

[0117] 6.2.3.4.3 Traversing F_n pq Compare the operation sequence in the relevant function code snippet, modify the relevant function code snippet of the target deep learning framework, and complete the F_n of the target framework. pq (i.e. missing function parameters). The method is:

[0118] 6.2.3.4.3.1 Traversing F_n pq The operation sequence seq_F_n pq (F_n pq It is FO i Elements of F_n pq Corresponding to one of the fo in 2.2.4.1.2.5 im , seq_F_n pq Corresponding to a seq_fo in 2.2.4.2.2.3 im ), according to seq_F_n pq Each operation in modifies the target framework function code snippet. The method is:

[0119] 6.2.3.4.3.1.1 Initialize r = 1;

[0120] 6.2.3.4.3.1.2 For seq_F_n pq The rth operation in seq_F_n pqr , if seq_F_n pqr The second element of the modified code key-value pair is empty, go to 6.2.3.4.3.1.2.1; if seq_F_n pqr The second element of the modified code key-value pair is non-empty, go to 6.2.3.4.3.1.2.2;

[0121] 6.2.3.4.3.1.2.1 According to the operation sequence seq_F_n pq The provided modification position key-value pairs (modification position, code line number) and seq_F_n pq The provided modification code key-value pair (source framework operator function code line, empty) maps the source framework operator function code snippet to the target framework operator function code snippet, and inserts the target framework operator function code snippet at the modification position. The method is:

[0122] 6.2.3.4.3.1.2.1.1 Read seq_F_n pq The source framework operator function code line source_code_F_n in the provided modification code key-value pair pq ;

[0123] 6.2.3.4.3.1.2.1.2 Source frame operator function code line source_code_F_npq Map to the target framework and get target_code_F_n pq The method is:

[0124] 6.2.3.4.3.1.2.1.2.1 Parsing source_code_F_n pq . Use Clang or Python's AST function library to parse source_code_F_n pq And generate an abstract syntax tree, identify the data type nodes (Clang's Type type node, Python's ast.Constant type node), exception type nodes (Clang's EnumConstantDecl type node) and tensor variable nodes (Clang's VarDecl node). If it is a data type node or an exception type node, go to 6.2.3.4.3.1.2.1.2.2; if it is a tensor variable node, go to 6.2.3.4.3.1.2.1.2.3.

[0125] 6.2.3.4.3.1.2.1.2.2 Use cosine similarity to find the most similar data type or exception type in the data type set or exception type set of the target deep learning framework (the data type set is usually located in the type.h file of the target deep learning framework, and the exception type set is located in the error.h file of the target deep learning framework), and parse the found type value into the corresponding data type node or exception type node, replacing source_code_F_n pq The original node in the image is mapped to the target framework after completion, and then go to 6.2.3.4.3.1.3;

[0126] 6.2.3.4.3.1.2.1.2.3 According to the tensor definition file of the target framework (usually located in Tensor.h of the target framework), make corresponding modifications to the tensor variable nodes, such as changing the calling method and initialization method of the tensor or updating related property settings, including device location, tensor shape, memory layout, gradient properties and data pointers. After completion, the above nodes are mapped to the target framework, and go to 6.2.3.4.3.1.3;

[0127] 6.2.3.4.3.1.2.2 According to the operation sequence seq_F_n pq The provided modification position key-value pairs (modification position, code line number) and seq_F_n pq The provided modification code key-value pair (source framework operator function code line, target framework operator function code line) is used to replace the F_n in the source framework operator function code line. pThe relevant code is added to the target frame operator function code line to obtain the modified target frame operator function code line, and the modified target frame operator function code line is inserted at the modified position, the method is:

[0128] 6.2.3.4.3.1.2.2.1 Read seq_F_n pq The source framework operator function code line source_code_F_n in the provided modification code key-value pair pq ; Read seq_F_n pq The target framework operator function code line target_code_F_n in the provided modification code key-value pair pq ;

[0129] 6.2.3.4.3.1.2.2.2 Parsing source_code_F_n pq . Use Clang or Python's ast function library to parse source_code_F_n pq In the abstract syntax tree format (called F_n pq _source_AST), traverse F_n pq _source_AST, identification and F_n p Nodes with the same name and their parent nodes.

[0130] 6.2.3.4.3.1.2.2.3 Parse and modify target_code_F_n pq , add F_n pq Related functions. Use Clang or Python's ast function library to parse target_code_F_n pq In the abstract syntax tree format (called F_n pq _target_AST), traverse F_n pq _target_AST, using variable name similarity matching (i.e., Levenshtein distance), find F_n pq The node in _target_AST that is most similar to the parent node identified in 6.2.3.4.3.1.2.2.2, and F_n p Node inserted into F_n pq Under the parent node of the node with the highest similarity in _target_AST. Complete F_n p Align related functions to get the modified target_code_F_n pq , go to 6.2.3.4.3.1.3;

[0131] 6.2.3.4.3.1.3 Let r=r+1. If r>R, then we get F_npq After alignment, go to 6.2.3.4.3.2 for the target deep learning framework function code snippet; if r≤R, go to 6.2.3.4.3.1.2;

[0132] 6.2.3.4.3.2 F_n pq The aligned target deep learning framework function code snippet imports the target framework image classification model and aligns the operator parameters. The method is:

[0133] 6.2.3.4.3.2.1F_n pq After alignment, the target deep learning framework function code snippet will have at most two segments (one segment is F_n pq The aligned target deep learning framework Python function code snippet; the other is F_n pq Aligned target deep learning framework C function code snippet), if the programming language type of the function code is Python, go to 6.2.3.4.3.2.2; if the programming language type of the function code is C, go to 6.2.3.4.3.2.3;

[0134] 6.2.3.4.3.2.2 F_n pq The aligned target deep learning framework Python function code snippet is encapsulated into a python function and imported into the target deep learning framework image classification model. The method is:

[0135] 6.2.3.4.3.2.2.1 F_n pq Aligned target deep learning framework Python function code snippet and F_n pq The relevant function code compares the context list in the snippet (i.e. the sub-functions in the same file being called, the functions, namespaces and header files to which the function code snippet belongs), and generates a list containing F_n pq The Python file of the aligned target deep learning framework Python function code snippet (the steps of building a Python file are: create a new Python file, write the header file, the function to which the function code belongs, and the sub-function under the same file into the Python file, and the function to which the function code belongs contains F_n pq The aligned target deep learning framework Python function code snippet, with the prefix "modified" added to the function name of the original function code), the sub-function under the same file, and the above content is written to the file).

[0136] 6.2.3.4.3.2.2.2 Use Python's import function (import statement) to import the Python file generated in 6.2.3.4.3.2.2.1 into the target framework image classification model and convert np The operator name is changed to the function name of the function to which the function code belongs.

[0137] 6.2.3.4.3.2.2.3 Determine F_n pq Does the target F_n exist in the aligned target deep learning framework function code snippet? pq Aligned target deep learning framework C function code snippet. If there is F_n pq If the aligned target deep learning framework C function code snippet exists, go to 6.2.3.4.3.2.3; if it does not exist, go to 6.2.3.4.4;

[0138] 6.2.3.4.3.2.3 F_n pq The aligned target deep learning framework C function code snippet is encapsulated into a C file, and the C file is built into a dynamic link library (.so or .dll) through the target framework CMakeList template uploaded by the user. The dynamic link library is imported into the target framework image classification model. The method is:

[0139] 6.2.3.4.3.2.3.1 F_n pq Aligned target deep learning framework C function code snippet and F_n pq The relevant function code compares the context list in the snippet (i.e. the sub-functions in the same file being called, the functions, namespaces and header files to which the function code snippet belongs), and generates a list containing F_n pq The C file of the aligned target deep learning framework C function code fragment (the steps of generating the C file are: creating a new C file, writing the header file, namespace declaration, the function to which the function code belongs, and the sub-function under the same file into the C file, and the function to which the function code belongs contains F_n pq The aligned target deep learning framework C function code snippet is added with the prefix "modified" to the function name of the original function code).

[0140] 6.2.3.4.3.2.3.2 Read the compilation content in the CMakeList template, modify the first parameter of the add_library function, specify it as a compilable C file, and obtain the modified CMakeList template;

[0141] 6.2.3.4.3.2.3.3 Use the operating system instruction cmake to compile the CMakeList template modified in 6.2.3.4.3.2.3.2 to obtain the dynamic link library of the function code;

[0142] 6.2.3.4.3.2.3.4 Use the operating system instruction LD_PRELOAD or LoadLibrary to link the dynamic link library of 6.2.3.4.3.2.3.3;

[0143] 6.2.3.4.4 Let q = q + 1. If q > Q, complete n p Frame mapping, get the node n after cross-frame mapping p ', n p 'Put it in NN'; go to 6.3; if q≤Q, go to 6.2.3.4.2;

[0144] 6.2.4 Construct an equivalent target framework function expression based on the operator function call sequence provided in SS. The method is:

[0145] 6.2.4.1n p is an element of SO, ordseq_n p The corresponding ordseq_so is found in step 2.2.5.4 i . According to the n provided in SS p The operator function call sequence ordseq_n p , process ordseq_n in order p For each element in , reconstruct the equivalent functional code in the target framework. The method is:

[0146] 6.2.4.1.1 Initialize s = 1;

[0147] 6.2.4.1.2 Determine ordseq_n ps If the operation type is a target frame operator, go to 6.2.4.1.2.1; if the operation type is a source operator parameter, go to 6.2.4.1.2.2; if the operation type is a basic arithmetic operator, go to 6.2.4.1.2.3.

[0148] 6.2.4.1.2.1 Get ordseq_n ps Operator name, go to 6.2.4.1.3;

[0149] 6.2.4.1.2.2 Get ordseq_n ps Parameter value (if there is no parameter value, get the default parameter value), go to 6.2.4.1.3;

[0150] 6.2.4.1.2.3 Get ordseq_n ps For the corresponding operators in the target framework, go to 6.2.4.1.3;

[0151] 6.2.4.1.3s Let s=s+1, if s>S, we get np If the equivalent functional code is reconstructed in the target framework, go to 6.2.4.2; if s≤S, go to 6.2.4.1.2;

[0152] 6.2.4.2 n p The equivalent functional code in the target framework is reconstructed to parse into a syntax abstract tree (called ordseq_n p _AST), ordseq_n p _AST root node as the node n after cross-frame mapping p ', n p 'Put it to NN'.

[0153] 6.3 Let p = p + 1. If p>P, complete the cross-frame mapping of NN and obtain NN' (including all the frame mapped nodes n p '), go to 6.4; if p≤P, go to 6.2;

[0154] 6.4 Replace NN in the source AST with NN' to form the grammatical abstract tree of the target framework image classification model (referred to as target AST). The method is:

[0155] 6.4.1 Initialize p = 1;

[0156] 6.4.2 According to the CC provided n p The position in the source AST, corresponding to n in NN' p 'Replace n p ;

[0157] 6.4.3 Let p = p + 1. If p>P, NN in the source AST has been replaced by NN', go to 6.4.4; otherwise, go to 6.4.2;

[0158] 6.4.4 Use the to_source function in Python's astor library to convert the target AST into the target framework image classification model;

[0159] 6.5 Send the target framework image classification model to the image classification model weight loading module.

[0160] In the seventh step, the image classification model weight loading module receives the target framework image classification model from the cross-frame mapping module and receives the source deep learning framework official model weights input by the user. Use the load function of the target framework image classification model to load the source deep learning framework official model weights, obtain the image classification inference model, and send the image classification inference model to the image classification module. The method is:

[0161] 7.1 Input the official model weights of the source deep learning framework into the state_dict function of the deep learning framework to obtain the official model weights in numpy format;

[0162] 7.2 Use the load function of the target framework image classification model to load the official model weights in numpy format into the target framework image classification model to obtain the image classification inference model;

[0163] 7.3 Send the image classification inference model to the image classification module.

[0164] In the eighth step, the image classification module receives the image classification inference model from the image classification model weight loading module, receives the image to be classified uploaded by the user, and outputs the classification result of the image. The method is:

[0165] The image classification module forward propagates to obtain the prediction result: the image to be classified uploaded by the user is input into the image classification inference model (the image is input in the form of RGB), and the image classification inference model outputs the classification result of the image to be classified.

[0166] The present invention can achieve the following beneficial effects:

[0167] 1. The first step of the present invention is to construct an image classification system based on cross-frame model conversion and operator difference alignment; the second to sixth steps realize cross-frame model conversion and align the operator differences in the intelligent model cross-deep learning framework conversion process through the cooperation of the operator difference analysis module, the operator difference mapping module, the model syntax parsing module, the operator analysis module, and the cross-frame mapping module; the seventh step is that the image classification model weight loading module can directly load the model weights in the official model library of the deep learning framework into the target framework image classification model obtained by the cross-frame conversion, which effectively solves the problems of being very time-consuming and complex to build and train accurate image classification models; and the official model weights are obtained by fully training the image classification model, which can ensure the image classification accuracy. Therefore, the time overhead of the present invention is limited to the cross-frame model conversion (the time overhead is in seconds), which is significantly reduced compared to training a new image classification model from scratch (the time overhead is in hours or even days).

[0168] 2. By testing the performance of 18 visual field reasoning models converted from PyTorch to PaddlePaddle, compared with the existing work (ONNX, a tool designed in "ONNX: Open Neural Network Exchange for Framework Interoperability" published by Bai X et al. at NeurIPS 2019, and PaConvert published by Baidu on Github), the present invention has stronger compatibility (the present invention supports all test models, while ONNX and PaConvert have operators that are not supported due to misalignment of operator differences), and the present invention effectively solves the operator differences in intelligent models. Compared with existing tools, it supports more models (up to 3.25 times). On the basis of adopting mature official model weights, the image classification stability is improved by 69.1% (measured by the standard deviation of the evaluation index) and the image classification accuracy is improved by up to about 4.5%, showing better compatibility and classification accuracy.

[0169] In summary, the present invention can effectively align the operator differences in the process of converting intelligent models across deep learning frameworks, which is superior to the existing image classification methods based on cross-frame model conversion and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0170] Figure 1 It is the overall flow chart of the present invention;

[0171] Figure 2 This is the logical structure diagram of the image classification method based on cross-frame model conversion and operator difference alignment constructed in the first step of the present invention. DETAILED DESCRIPTION

[0172] The present invention will be described below in conjunction with the accompanying drawings.

[0173] like Figure 1 As shown, the present invention comprises the following steps:

[0174] The first step is to build an image classification system based on cross-frame model conversion and operator difference alignment. Figure 2 As shown, it consists of an operator difference analysis module, an operator difference mapping module, a model syntax parsing module, an operator analysis module, a cross-frame mapping module, an image classification model weight loading module, and an image classification module;

[0175] The operator difference analysis module is an operator difference extraction tool, which is connected to the operator difference mapping module. It compares and analyzes the source deep learning framework operator list SO and the target deep learning framework operator list TO input by the user, extracts the differences in the operator interface and operator function code of the operators corresponding to SO and TO, and obtains the operator name, operator interface mapping relationship, operator difference type, operator function code comparison fragment and operator function call sequence. The operator name is a string type data, which stores the source deep learning framework operator name. The operator interface mapping relationship is a dictionary type data, including two key-value pairs (a key-value pair is a tuple consisting of a key and a value), namely the operator name mapping relationship key-value pair (source deep learning framework operator name, target deep learning framework operator name), parameter mapping relationship key-value pair (key storage source deep learning framework operator parameters, target deep learning framework operator parameters); the operator difference type is a string type data, including three types, namely "operator function signature and usage differences", "operator behavior differences" and "operator missing differences"; the operator function code comparison fragment is a hierarchical dictionary, the root level contains a key-value pair (the associated source deep learning framework parameters in the function code, the modification sequence). The modification sequence (array type) contains multiple modification operations, each modification operation describes the specific modification made to the source framework operator function code, and each modification operation is a sub-dictionary containing the following three fields: programming language type (string type, such as: Python, C), function code fragment context information (dictionary type), operation list (containing multiple specific modification operations, each operation is a dictionary type). The function code snippet context information contains three key-value pairs: sub-function call key-value pair ("sub-function call", sub-function call list), header file key-value pair ("header file", header file code), namespace key-value pair ("namespace", the namespace of the source framework operator function code). The operation list contains multiple operations, each of which contains two key-value pairs: modification position key-value pair ("modification position", the line number of the code in the function where the function code is located), modification code key-value pair (source framework function code snippet, target framework function code snippet). Each element in the operator function call sequence is an operation, corresponding to one of the target framework operator, source framework operator parameter, and basic operator. The operator name, operator difference type, operator interface mapping relationship, operator function code comparison snippet, and operator function call sequence are sent to the operator difference mapping module.

[0176] The operator difference mapping module is connected to the operator difference analysis module, the model syntax parsing module, and the cross-frame mapping module, and receives the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment, and operator function call sequence from the operator difference analysis module. Map the above content into the operator mapping table SS. SS is a hierarchical dictionary with a root level of a key-value pair list. In each key-value pair in the key-value pair list, the key is the operator name and the value is a dictionary. It contains two or three key-value pairs according to the operator difference type: when the operator difference type is "operator function signature and usage difference", the value contains the operator difference type and operator interface mapping relationship key-value pair ("operator interface mapping relationship", operator interface mapping relationship); when the operator difference type is "operator behavior difference", the value contains the operator difference type, operator interface mapping relationship key-value pair, operator function code comparison fragment key-value pair ("operator function code comparison fragment", storage operator function code comparison fragment); when the operator difference type is "operator missing difference", the value contains the operator difference type key-value pair, operator interface mapping relationship key-value pair, operator function call sequence key-value pair ("operator function call sequence", operator function call sequence). Send the operator mapping table SS to the model syntax parsing module and the cross-framework mapping module.

[0177] The model syntax parsing module is connected to the operator difference mapping module, the operator analysis module, the cross-frame mapping module, and the image classification model weight loading module. It receives the source frame image classification model uploaded by the user, receives the operator mapping table SS from the operator difference mapping module, obtains the operator name O and the source frame image classification model uploaded by the user from SS, and parses the source frame image classification model uploaded by the user into a source abstract syntax tree (referred to as source AST). An abstract syntax tree is a grammatical structure that expresses a model in a tree structure. In this structure, each node represents a basic component (operator, function, module) in the image classification model, and the relationship between nodes represents the hierarchy and dependency of these components. The source AST represents the components of the image classification model in an abstract way that is independent of the framework. The model syntax parsing module identifies and marks the nodes related to operators in the source AST, filters out the framework-related node set NN from the source AST through the framework root directory name (such as "torch" for PyTorch, "paddle" for Paddle, "tensorflow" for TensorFlow, etc.), sends NN and source AST to the operator analysis module, and sends NN to the cross-framework mapping module.

[0178] The operator analysis module is connected to the model syntax parsing module and the cross-frame mapping module. It receives the frame-related node set NN and the source AST from the model syntax parsing module, analyzes each node in NN, and obtains the context information CC of each node in NN in the source AST. The context information includes the operator name, operator parameter list (each element includes parameter name, parameter value and parameter default value), node type (ast.Call or ast.attribute, ast is a standard library of Python), node position (that is, the code line where the node is located), and pointers to parent nodes and child nodes. CC and the source AST are sent to the cross-frame mapping module.

[0179] The cross-frame mapping module is connected to the operator difference mapping module, the model syntax parsing module, the operator analysis module, and the image classification model weight loading module. It receives the operator mapping table SS from the operator difference mapping module, receives the framework-related node set NN from the model syntax parsing module, receives the operator's context information CC and source AST from the operator analysis module, and receives the target framework CMakeList template uploaded by the user. It performs cross-frame mapping on each node in NN, and performs different difference alignment operations according to the operator difference type of each node in NN corresponding to the operator in SS. The alignment process needs to complete the operator interface mapping (using the operator interface mapping relationship in SS to complete the operator API mapping), the target framework operator Python function code alignment (encapsulate the aligned Python function code into a library function, and import it into the target framework image classification model to replace the source framework operator), and the target framework operator C function code alignment (use the CMakeList template to compile the C function code into a .so file, and link these .so files in the target framework image classification model to achieve C function code alignment). Finally, the mapped frame-related node set NN' is obtained, and according to the node position in the operator context CC, the nodes in NN in the source AST are replaced with the corresponding nodes in NN' to obtain the abstract syntax tree of the target frame (referred to as target AST), and the target AST is reparsed into the target frame image classification model, and the target frame image classification model is sent to the image classification model weight loading module.

[0180] The image classification model weight loading module is connected to the cross-frame mapping module and the image classification module. It receives the official model weights of the source deep learning framework uploaded by the user, receives the target framework image classification model from the cross-frame mapping module, converts the format of the official model weights of the source deep learning framework and loads them into the target framework image classification model to obtain the image classification inference model, and sends the image classification inference model to the image classification module.

[0181] The image classification model module is connected to the image classification model weight loading module, receives the image to be classified uploaded by the user, receives the image classification inference model from the image classification model weight loading module, inputs the image to be classified uploaded by the user into the image classification inference model, and obtains the final classification result of the image to be classified.

[0182] In the second step, the operator difference analysis module compares the operator list SO of the source deep learning framework and the operator list TO of the target deep learning framework, and sends the analysis results, i.e., the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence, to the operator difference mapping module. The method is:

[0183] 2.1 Operator difference analysis module receives the operator list SO of the source deep learning framework and the operator list TO of the target deep learning framework input by the user, SO = {so 1 ,so 2 ,…,so i ,…,so I}, so i is the i-th operator in SO, I is the total number of operators in the source deep learning framework, 1≤i≤I; TO={to 1 ,to 2 ,…,to j ,…,to J}, to j is the jth operator in TO, J is the total number of operators in the target deep learning framework, 1≤j≤J;

[0184] 2.2 Traverse SO, find the matching operator of each operator to be analyzed in SO in TO, and analyze the function codes of the two matching operators to obtain the function code comparison fragments of the two matching operators. The method is:

[0185] 2.2.1 Construct an empty list NAME_TO and put to 1 ,to 2 ,…,to j ,…,to J Place it in NAME_TO.

[0186] 2.2.2 Initialize i=1;

[0187] 2.2.3 For the i-th operator so in SO i , will so iand NAME_TO are input to the get_close_matches function in Python's difflib library. The function of the get_close_matches function is to find several strings that are most similar to the target string in a string list. The get_close_matches function is based on so i The operator name in NAME_TO is searched for an operator name that matches the operator name in NAME_TO, and a list containing all the operator names in NAME_TO that match the operator name in NAME_TO is obtained. i List of approximate matching operator names LL i If LL i Not empty, execute 2.2.3.1; if LL i If it is empty, execute 2.2.3.2;

[0188] 2.2.3.1 Find LL in TO i The first element of finds the corresponding target frame operator in TO, let to j , to j for i Matching operator, go to 2.2.4;

[0189] 2.2.3.2 Explanation: TO does not exist with so i Match the target frame operator, go to 2.2.5.

[0190] 2.2.4 At this time, TO exists with so i The target frame operator to match j , analyze so i and to j The function code is obtained i and to j Function code comparison snippet, the method is:

[0191] 2.2.4.1 Generate so i and to j The interface mapping relationship is as follows:

[0192] 2.2.4.1.1 Record so i and to j Operator name, formed so i and to j Operator name mapping key-value pairs (so i The operator name, to j operator name).

[0193] 2.2.4.1.2 Initialize so i Parameter Mapping List SL i and the failed mapping parameter list fo i Is empty. Traverse soi Operator parameter list param_so i ,param_so i = {param_so i1 ,param_so i2 ,…,param_so ik ,…,param_so iK}, param_so ik param_so i The kth parameter, K is the parameter in param_so i The total number of parameters, 1≤k≤K. Set the operator parameter name param_so ik As a keyword, in to j The parameter list param_to j Search in param_to j = {param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL}, L is param_to j The total number of parameters in ,1≤l≤L. That is, param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL with param_so ik Perform name matching in to j Find the value corresponding to param_so ik The corresponding operator parameters. The specific method is:

[0194] 2.2.4.1.2.1 Initialize k = 1;

[0195] 2.2.4.1.2.2 Traversing to j The parameter list param_to j , param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL with param_so ik To match, use:

[0196] 2.2.4.1.2.2.1 Initialize l = 1;

[0197] 2.2.4.1.2.2.2 If param_soik and param_to jl The names are the same, or their prefixes or suffixes partially match (for example, param_so ik Starting with "eps", param_to jl Starts with "epslion" or param_so ik Ends with "Data", param_to jl ends with "SensorData"), then param_so ik and param_to jl Matching is successful. If the match is successful, a key-value pair (param_so ik , param_to jl ), go to 2.2.4.1.2.3; if that fails, go to 2.2.4.1.2.2.3;

[0198] 2.2.4.1.2.2.3param_so ik and param_to jl Matching failed, set l = l + 1. If l>L, go to 2.2.4.1.2.4; if l≤L, go to 2.2.4.1.2.2.2;

[0199] 2.2.4.1.2.3 At this time param_so ik and param_to jl Matching is successful, (param_so ik , param_to jl ) Add parameter mapping list SL i , let k=k+1, if k>K, go to 2.2.4.1.2.5; if k≤K, go to 2.2.4.1.2.2;

[0200] 2.2.4.1.2.4 Explanation of param_so at this time ik With the target frame operator to j If all parameters in the match fail, param_so ik Add param_so i Failed mapping parameter list FO i , go to 2.2.4.1.2.5;

[0201] 2.2.4.1.2.5 Now complete so i All parameters match, forming so i Parameter Mapping List SL i and so i Failed to map parameter list fo i , SLi ={sl i1 ,sl i2 ,…,sl it ,…,sl iT}, sl it For SL i The tth so i and to j Parameter mapping key-value pairs, T is so i and to j The total number of parameter mapping key-value pairs; FO i ={fo i1 ,fo i2 ,…,fo im ,…,fo iM}, for im For FO i The mth mapping failure parameter in , M is FO i The total number of all failed parameters in;

[0202] 2.2.4.1.3 Build the operator interface mapping relationship dictionary. The operator interface mapping relationship dictionary contains two key-value pairs, one for the so formed in step 2.2.4.1.1 i and to j The operator name mapping key-value pair and the parameter mapping list key-value pair ("parameter mapping list", SL i ), the dictionary is the operator interface mapping relationship;

[0203] 2.2.4.2 Traversing FO i , follow up i1 ,fo i2 ,…,fo im ,…,fo iM Related operator implementation codes, obtain operator function code comparison snippets (that is, track and record the code lines containing the parameter), the method is:

[0204] 2.2.4.2.1 Initialize m = 1;

[0205] 2.2.4.2.2 Use PDB (Python debugger) and GDB (GNU debugger) to run the operator so i and to j Among them, PDB is used to debug the code modules written in Python, providing functions such as step-by-step execution, breakpoint setting, and variable inspection. im Related Python function code comparison snippets; GDB is used to debug low-level code written in C / C++, supporting memory inspection, thread debugging and performance analysis, and obtaining fo imRelated C function code comparison snippets. Through this joint debugging method, comprehensive records and fo im Related code running status. Merge fo im Related Python function code comparison snippets and fo im Compare the relevant C function code snippets to get the complete build fo im The relevant function code comparison snippet is as follows:

[0206] 2.2.4.2.2.1 Build fo im Related Python function code comparison snippets: First, use PDB to debug the operator so i and to j , set breakpoints in the __init__ and forward functions in PDB. Then run so i When the execution reaches the breakpoint, it enters debug mode. In debug mode, PDB uses the "enter function" instruction and the "next line" instruction to track the so i Python function code, execute the code line by line, and record so i fo im Related function code (in code line format). For each line so i fo im Related function code, run to j When the execution reaches the breakpoint, it enters the debug mode. In debug mode, PDB uses the "enter function" instruction and the "next line" instruction to track to j Python function code, execute the code line by line, use Python's ast parse function to parse so i fo im The relevant lines of code and to j The current debugging code line uses the tree isomorphism (AHU) algorithm to determine so by matching the node labels and the order and connection relationship of their child nodes. i fo im The subtree structure of the relevant code lines is the same as to j The similarity of the subtree of the currently debugged code line in the local structure is obtained to obtain the similarity score between the two. i fo im The subtree structure of the relevant code lines is the same as to j If the subtree structure of the currently debugged code line matches (similarity score greater than 0.8), then build so i fo im The Python function code sends snippet key-value pairs (so i Function code lines, to j Function code line); if toj Does not exist and so i fo im Related lines of code similar to the code, also build so i fo im Related Python function code snippet key-value pairs (so i function code lines, empty). Build fo im A list of related Python function code contexts, including the called sub-functions in the same file, the function to which the function code snippet belongs, the namespace and header file. im Related Python function code comparison snippets, fo im The relevant Python function code comparison snippet uses a dictionary data type, which contains three key-value pairs: programming language type key-value pair ("programming language type", "Python"), context information key-value pair ("context information", fo im Related Python function code context list), function code list key-value pairs ("function code key-value pair list", so i fo im Related Python function code comparison snippet key-value pair list).

[0207] 2.2.4.2.2.2 Construction of fo im Related C function code comparison snippet: Using GDB to debug the operator so i and to j , extract the function call stack (actually a list) of the operator program stack in GDB, use the keyword list (impl, functor, kernel) to perform suffix matching (that is, the function name ends with the keyword), filter out the core calculation function (kernel function), and set a breakpoint for the kernel function. In debug mode, run so i When the execution reaches the breakpoint, GDB uses the "enter function" instruction and the "next line" instruction to track the so i C function code, execute the code line by line, and record so i fo im Related function code (in code line format). For each line so i fo im Related function code, run to j When the execution reaches the breakpoint, it enters the debug mode. In debug mode, GDB uses the "enter function" instruction and the "next line" instruction to track to jC function code, using Clang (Clang is a powerful C / C++ / Objective-C compiler front end, which can not only compile code, but also generate a detailed abstract syntax tree (AST)) to parse so i fo im Parameter related code lines and to j The current debugged code line uses the tree isomorphism algorithm to determine the similarity of the two trees in local structure by matching the order and connection relationship of node labels and their child nodes, and obtain the similarity score between the two. i fo im The subtree structure of the relevant code lines is the same as to j If the subtree structure of the currently debugged code line matches (similarity score greater than 0.8), then build so i fo im C function code snippet key-value pairs (so i The C function code line, to j C function code line); if to j Does not exist and so i fo im Related lines of code similar to the code, also build so i fo im Related C function code snippet key-value pairs (so i lines of C function code, empty). Build fo im A list of related C function code contexts, including the called sub-functions in the same file, the function to which the function code snippet belongs, the namespace and header file. im Related C function code comparison snippet (dictionary type), fo im The relevant C function code comparison snippet contains three key-value pairs: programming language type key-value pair ("programming language type", "C"), context information key-value pair ("context information", fo im Related C function code context list), function code list key-value pairs ("function code key-value pair list", so i fo im Related C function code comparison snippet key-value pair list).

[0208] 2.2.4.2.2.3 Merge fo im Related Python function code comparison snippets and fo im Compare the relevant C function code snippets to get the complete build fo im Related functional code comparison snippets, complete build fo im The relevant function code comparison snippet contains two key-value pairs: the associated parameter key-value pair ("associated parameter", foim parameter name), operation sequence key-value pairs ("operation sequence", fo im Related Python function code comparison snippets and fo im The dictionary list formed by combining the relevant C function code comparison fragments). im Related Python function code comparison snippets and fo im The relevant C function code comparison snippet contains the function code list key-value pairs, which are defined as fo im The operation sequence seq_fo im , seq_fo im ={seq_fo im1 ,seq_fo im2 ,…,seq_fo imr ,…,seq_fo imR}, seq_fo imr For im The rth function code comparison fragment key-value pair, R is so i fo im Related Python function code comparison snippet key-value pair list and so i fo im The relevant C function code compares the sum of the fragment key-value pair lists).

[0209] 2.2.4.2.3 Let m = m = +1. If m>M, change fo i1 ,fo i2 ,…,fo im ,…,fo iM The relevant functional code is compared and merged to obtain FO i Related function code comparison fragment (dictionary list type), go to 2.2.6; if m≤M, go to 2.2.4.2.2.1;

[0210] 2.2.5 At this time, TO does not exist with so i Matching target frame operator, analysis so i The kernel function of i The operator function call sequence is:

[0211] 2.2.5.1 Using GDB to debug so i , extract the function call stack (actually a list) of the operator program heap in GDB, use framework-related keywords (impl, functor, kernel) to query so i Perform suffix matching on the function calls in the function call stack (i.e., the function call name ends with a keyword), and filter out soi kernelfunction in the function call stack.

[0212] 2.2.5.2 Use the function syntax analysis method to analyze the core computing function (kernel function) and form the operation sequence of the kernel function according to the actual execution order of the control flow. The method is:

[0213] 2.2.5.2.1 Use Clang to generate an abstract syntax tree for the kernel function and obtain the AST of the kernel function.

[0214] 2.2.5.2.2 Analyze the AST of the kernel function, process the return statement of the kernel function (Clang's returnStmt type will mark the return statement), and record the code line where the return statement is located.

[0215] 2.2.5.2.3 Initialize an empty operation sequence list corresponding to the kernel function, which is used to record all relevant operations in the analysis process (target framework operator, source framework operator parameters, sub-functions called in the same file, arithmetic operators).

[0216] 2.2.5.2.4 Use the syntax expression analysis method to analyze the return statement code of the kernel function and obtain the syntax expression of the return statement. The return statement corresponds to the returnStmt node in the AST of the kernel function, which contains a return value expression (the expression is a tree structure). Analyze and identify the local variables (clang's VarDecl node), basic operators (such as addition, subtraction, multiplication, division, etc., clang's UnaryOperator node, clang's BinaryOperator node, clang's ConditionalOperator node, clang's CompoundAssignOperator node), operator parameters (i.e., kernel function parameters, clang's ParmVarDecl node), and function calls (clang's CallExpr node).

[0217] 2.2.5.2.5 Use the depth-first search method to traverse the syntax expression of the return statement in 2.2.5.2.4 to obtain the operation sequence of the kernel function.

[0218] 2.2.5.3 Use the local variable analysis method to analyze all local variables (i.e. VarDecl nodes) in the kernel function's operation sequence. Trace back the assignment process of local variables, obtain the syntax expression of local variables and replace the syntax expression with the VarDecl node in the operation sequence to obtain the kernel function operation sequence that does not contain the VarDecl node. The method is:

[0219] 2.2.5.3.1 Analyze the assignment statement of the local variable (AssignExpr node), the local variable on the left side of the assignment statement, and a syntax expression on the right side of the assignment statement. Use the syntax expression analysis method described in 2.2.5.2.4 to analyze the right side of the assignment statement, analyze the local variables, basic operators, operator parameters, and function calls therein, use the depth-first search method to build the operation sequence of the AssignExpr node syntax expression, and replace the VarDecl node currently being analyzed in the operation sequence of the kernel function.

[0220] 2.2.5.3.2 If there are no local variables (i.e., VarDecl nodes) in the operation sequence of the kernel function, complete the local variable analysis in the return statement to obtain the kernel function operation sequence that does not contain VarDecl nodes, and go to 2.2.5.4; if there are still other VarDecl nodes in the operation sequence of the kernel function, let the local variables currently being analyzed be other VarDecl nodes, and go to 2.2.5.3.1; go to 2.2.5.3.1.

[0221] 2.2.5.4 Use the function call analysis method to further analyze all function calls (i.e., CallExpr nodes) in the kernel function's operation sequence. Each function call operation corresponds to a CallExpr node. Get the name of the function call of the CallExpr node and the parameter list in the function call. Suffix match the function call name with the target framework operator name. If the function call of the CallExpr node matches the target framework operator name successfully, replace the function call of the CallExpr node with the target framework operator; if the function call of the CallExpr node does not match the target framework operator name successfully, then deeply analyze the function call code to form the operation sequence of the CallExpr node, replace the CallExpr node in the kernel function's operation sequence, and get so i The operator function call sequence ordseq_so i The method is:

[0222] 2.2.5.4.1 Get the function name and parameter list of each CallExpr node in the kernel function's operation sequence;

[0223] 2.2.5.4.2 Match the relationship between each CallExpr node in the kernel function's operation sequence and the target framework operator. Check whether the function name in the CallExpr node matches the operator name in the target framework by exact matching or suffix matching (for example, whether the function matrixMultiply matches the operator MatrixMul in the target framework. In addition to matching the function name, it is also necessary to confirm whether the function's parameter type and quantity meet the requirements of the target framework operator). If the function name and parameters match successfully, replace the CallExpr node with the operator of the target framework and go to 2.2.5.4.4; if not, further analyze the CallExpr node and go to 2.2.5.4.3;

[0224] 2.2.5.4.3 Use the function syntax analysis method of 2.2.5.2 to analyze the function call content (i.e., function code) corresponding to the CallExpr node, and form an operation sequence of the CallExpr node according to the actual execution order of the control flow; use the local variable analysis method of 2.2.5.3 to analyze all VarDecl nodes (local variables) in the operation sequence of the CallExpr node, backtrack the assignment process of the local variables, obtain the syntax expression of the local variables and replace the VarDecl node in the operation sequence with the syntax expression to obtain the operation sequence of the CallExpr node that does not contain the VarDecl node; replace the CallExpr node currently being analyzed in the operation sequence of the kernel function with the operation sequence of the CallExpr node that does not contain the VarDecl node.

[0225] 2.2.5.4.4 Continue to traverse the CallExpr nodes in the kernel function's operation sequence. If there is an unanalyzed CallExpr node, go to 2.2.5.4.1; if not, get so i The operator function call sequence ordseq_so i , ordseq_so i ={ordseq_so i1 ,ordseq_so i2 ,…,ordseq_so is ,…,ordseq_so iS}, ordseq_so is is ordseq_so iThe sth operation, S is ordseq_so i The total number of operations in the sequence, go to 2.2.6;

[0226] 2.2.6 Let i = i + 1. If i>I, go to 2.3; if i≤I, go to 2.2.3;

[0227] 2.3 will so i The operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment or operator function call sequence are sent to the operator difference mapping module.

[0228] In the third step, the operator difference mapping module receives the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence from the operator difference analysis module, traverses the source deep learning framework operator list SO, and processes SO according to different operator difference types. 1 ,so 2 ,…,so i ,…,so I The operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence are mapped into the operator mapping table SS, and the SS is sent to the model syntax parsing module and the cross-frame mapping module. The method is:

[0229] 3.1 Initialize the operator mapping table SS to be empty, and initialize i=1;

[0230] 3.2 According to so i The operator difference type is handled differently: if the operator difference type is "difference in operator function signature and usage", jump to 3.2.1; if the operator difference type is "difference in operator behavior", jump to 3.2.2; if the operator difference type is "difference in operator missingness", jump to 3.2.3;

[0231] 3.2.1 Construct a hierarchical dictionary to record three key-value pairs: operator name key-value pair ("operator name", so i operator name), operator difference type key-value pair ("operator difference type", so i operator difference type), operator interface mapping key-value pairs ("operator interface mapping", so i operator interface mapping relationship); add the hierarchical dictionary to SS and go to 3.3;

[0232] 3.2.2 Construct a hierarchical dictionary to record four key-value pairs: operator name key-value pair ("operator name", so i operator name), operator difference type key-value pair ("operator difference type", so ioperator difference type), operator interface mapping key-value pairs ("operator interface mapping", so i operator interface mapping relationship), operator function code comparison fragment key-value pairs ("operator function code comparison fragment", so i ); Add the hierarchical dictionary to SS and go to 3.3;

[0233] 3.2.3 Build a hierarchical dictionary to record three key-value pairs: operator name key-value pair (key is "operator name", so i operator name), operator difference type key-value pair ("operator difference type", so i operator difference type), operator function call sequence key-value pairs ("operator function call sequence", so i operator function call sequence); add the hierarchical dictionary to SS and go to 3.3;

[0234] 3.3 Let i=i+1. If i>I, it means that SO is processed and SS is obtained, go to 3.4; if i≤I, go to 3.2;

[0235] 3.4 Send SS to the model syntax parsing module and cross-frame mapping module.

[0236] In the fourth step, the model syntax parsing module receives the source frame image classification model uploaded by the user, and receives the operator difference mapping table SS from the operator difference mapping module. The source frame image classification model is converted into AST (referred to as source AST), and the components of the source frame image classification model are represented in a frame-independent manner. The nodes related to the operator are identified and marked, and the frame-related node set NN is filtered out using the root directory name of the frame, and NN is sent to the operator analysis module and the cross-frame mapping module. The method is:

[0237] 4.1 Parse the source framework image classification model into a syntax abstract tree (source AST for short); Use the parse function in the Python standard library ast to convert the source framework image classification model code into AST.

[0238] 4.2 Use the visit function of the source AST to recursively traverse the source AST nodes, identify each Call type or Attribute type node in the source AST, filter out the framework-related nodes, and obtain the related node set NN. The method is:

[0239] 4.2.1 Start traversing from the root node of the source AST, and let the current node v = the root node of the source AST;

[0240] 4.2.2 Determine the node type of v. If the node type of v is a Call type or an Attribute type, go to 4.2.2.2. If the node type of v is neither a Call type nor an Attribute type, go to 4.2.2.1.

[0241] 4.2.2.1 does not mark v and goes to 4.2.2.3;

[0242] 4.2.2.2 Get the operator name in v and traverse so in SS i The operator name of is matched with the operator name in v. If the match is successful, v is determined to be a framework-related node and recorded. The method is:

[0243] 4.2.2.2.1 Initialize the relevant node set NN to be empty, and initialize i=1;

[0244] 4.2.2.2.2 Get so from SS i The operator name of is matched against the operator name of v (obtained by getting the value of v). i If the operator name of and the operator name of v match successfully, the node is marked and v is placed in NN, and the process goes to 4.2.2.3. If the match fails, no marking is performed and the process goes to 4.2.2.2.3.

[0245] 4.2.2.2.3 Let i = i + 1. If i > I, go to 4.2.2.3. If i ≤ I, go to 4.2.2.2.2.

[0246] 4.2.2.3 Traverse the next node of the source AST (call the visit function of AST, which will automatically recursively visit all nodes). If the next node exists, go to 4.2.2; if the next node does not exist, it means that the framework-related node set NN containing all marked nodes is obtained. NN = {n 1 ,n 2 ,…,n p ,…,n P},n p is the p-th frame-related node (operator) of the NN, that is, v, n placed in the NN p It has three attributes: name (operator name), args (operator parameters), and class (node ​​type corresponding to the operator in the source AST). 1≤p≤P, where P is the number of framework-related nodes in the NN. Go to 4.3.

[0247] 4.3 Send NN and source AST to the operator analysis module and cross-frame mapping module.

[0248] In the fifth step, the operator analysis module receives the frame-related node set NN and the source AST from the model syntax parsing module, parses the context information CC of the P frame-related nodes in NN in the source AST, and sends CC to the cross-frame mapping module.

[0249] The method is:

[0250] 5.1 Let p = 1; initialize the context information CC in the source AST to be empty, and initialize the node context information list CC p Is empty.

[0251] 5.2 Get the nth frame-related node n in NN p The context information of the source AST includes: operator name, operator parameter list, node type, node position (the code line where the node is located), and pointers to parent nodes and child nodes. Each parameter in the operator parameter list includes: parameter name, parameter value, and parameter default value. The method is:

[0252] 5.2.1 Use the visit function of the source AST to traverse the source AST and record n p A list of parent nodes and child nodes;

[0253] 5.2.2 Access n p Attribute name, get n p The name of the operator;

[0254] 5.2.3 Access n p The attribute args, get n p The parameter list.

[0255] 5.2.4 Access n p The attribute __class__ of p Node type;

[0256] 5.2.5 Using astor.to_source function to obtain n p The code lines of the node in the source framework image classification model (astor is a standard function library of Python);

[0257] 5.3 replace 5.2 with n p The operator name, operator parameter list, node type, node position, and pointers to parent nodes and child nodes are added to CC p In the p Join CC.

[0258] 5.4 Let p = p + 1. If p > P, go to 5.5. If p ≤ P, go to 5.2.

[0259] 5.5 Send CC to the cross-frame mapping module.

[0260] In the sixth step, the cross-frame mapping module receives SS from the operator difference mapping module, receives NN from the model syntax parsing module, receives CC and source AST from the operator analysis node module, receives the target framework CMakeList template uploaded by the user, traverses the source AST, performs cross-frame mapping, and 1 ,n 2 ,…,n p ,…,n P Combine CC and SS to align n 1 ,n 2 ,…,n p ,…,n P The operator differences in the cross-deep learning framework mapping process are used to obtain the framework-related node set NN' after cross-frame mapping, and NN' is used to replace NN in the source AST to obtain the grammatical abstract syntax tree of the target framework (referred to as target AST). The target AST is parsed into the target framework image classification model, and the target framework image classification model is sent to the image classification model weight loading module. The method is:

[0261] 6.1 Let p = 1; initialize the frame-related node set NN' after cross-frame mapping to be empty;

[0262] 6.2 According to n p The operator difference type in the operator difference mapping table SS is used to perform cross-frame mapping and align the operator differences in the cross-frame mapping process to obtain the cross-frame mapping node n. p '. The method is:

[0263] 6.2.1 If the operator difference type is "operator function signature and usage differences", go to 6.2.2; if the operator difference type is "operator behavior differences", go to 6.2.3; if the operator difference type is "operator missing differences", go to 6.2.4;

[0264] 6.2.2 If the operator difference type is "Operator function signature and usage differences", complete n p Interface mapping, get n after cross-frame mapping p '. The method is:

[0265] 6.2.2.1 Reading n in SS p Operator interface mapping relationship, modify n according to the operator name in the mapping relationship p Name, complete n p Operator name mapping;

[0266] 6.2.2.2 Access n p The args attribute (i.e. n p Parameter list), let n p The parameter list is param_np , param_n p = {param_n p1 ,param_n p2 ,…,param_n po ,…,param_n pO}, param_n po param_n p The oth parameter of p The total number of parameters actually used, 1≤o≤O.

[0267] 6.2.2.3 Access n p __class__. attribute, if the attribute is "attribute", go to 6.3; if the attribute is "Call", go to 6.2.2.4;

[0268] 6.2.2.4 Using the operator parameter mapping method, according to n in SS p Parameter mapping list SL of interface mapping relationship i param_n p After the mapping is completed, the type n p (i.e. the operator type is "difference in operator function signature and usage") can directly get n p ', n p 'Put it to NN'. The method is:

[0269] 6.2.2.4.1 Initialize o = 1;

[0270] 6.2.2.4.2 For the oth param_n po , extract n from SS p Parameter mapping list SL in the interface mapping relationship i , find SL i to param_n po As a key-value pair, param_n po Replace with the corresponding value in the key-value pair;

[0271] 6.2.2.4.3 Let o = o + 1. If o > O, then n is completed. p Interface mapping, obtain node n after cross-frame mapping p ', go to 6.2.2.5; if o≤O, go to 6.2.2.4.2;

[0272] 6.2.2.5 n p 'Put to NN', go to 6.3;

[0273] 6.2.3 For n pPerform operator interface mapping and operator parameter function alignment. The method is:

[0274] 6.2.3.1 Read n p Operator interface mapping relationship, modify n according to the operator name in the mapping relationship p Name, complete n p Operator name mapping;

[0275] 6.2.3.2 Access n p The args attribute (n p .args is n p Parameter list), let n p The parameter list is param_n p .

[0276] 6.2.3.3 Using the operator parameter mapping method of 6.2.2.4, according to n in SS p Parameter mapping list SL of interface mapping relationship i param_n p After the mapping is completed, the type n p (That is, the operator type is "Operator Behavior Difference") Only some parameters are mapped successfully, and some parameters fail to be mapped.

[0277] 6.2.3.4 Collect the failed mapping parameters in 6.2.3.3 and get F_n p , F_n p ={F_n p1 ,F_n p2 ,…,F_n pq ,…,F_n pq}, F_n pq n p The qth failed mapping parameter of , where Q is the source frame operator n p Total number of failed mapping parameters, n p Corresponding to an element in SO, let the element be so i , F_n p So i Failed to map parameter list fo i For F_n p , using n in S p Operator function code comparison snippet, traversing FO i , align F_n p Related function codes, get the node n after cross-frame mapping p '. The method is:

[0278] 6.2.3.4.1 Initialize q = 1;

[0279] 6.2.3.4.2 Find FO i Chinese and Japanese F_n pq The key-value pair as the key. If it exists, the value of the key-value pair is F_n pq For the relevant function code comparison snippet, go to 6.2.3.4.3; if it does not exist, go to 6.2.3.4.4;

[0280] 6.2.3.4.3 Traversing F_n pq Compare the operation sequence in the relevant function code snippet, modify the relevant function code snippet of the target deep learning framework, and complete the F_n of the target framework. pq (i.e. missing function parameters). The method is:

[0281] 6.2.3.4.3.1 Traversing F_n pq The operation sequence seq_F_n pq (F_n pq It is FO i Elements of F_n pq Corresponding to one of the fo in 2.2.4.1.2.5 im , seq_F_n pq Corresponding to a seq_fo in 2.2.4.2.2.3 im ), according to seq_F_n pq Each operation in modifies the target framework function code snippet. The method is:

[0282] 6.2.3.4.3.1.1 Initialize r = 1;

[0283] 6.2.3.4.3.1.2 For seq_F_n pq The rth operation in seq_F_n pqr , if seq_F_n pqr The second element of the modified code key-value pair is empty, go to 6.2.3.4.3.1.2.1; if seq_F_n pqr The second element of the modified code key-value pair is non-empty, go to 6.2.3.4.3.1.2.2;

[0284] 6.2.3.4.3.1.2.1 According to the operation sequence seq_F_n pq The provided modification position key-value pairs (modification position, code line number) and seq_F_n pq The provided modification code key-value pair (source framework operator function code line, empty) maps the source framework operator function code snippet to the target framework operator function code snippet, and inserts the target framework operator function code snippet at the modification position. The method is:

[0285] 6.2.3.4.3.1.2.1.1 Read seq_F_n pq The source framework operator function code line source_code_F_n in the provided modification code key-value pair pq ;

[0286] 6.2.3.4.3.1.2.1.2 Source frame operator function code line source_code_F_n pq Map to the target framework and get target_code_F_n pq The method is:

[0287] 6.2.3.4.3.1.2.1.2.1 Parsing source_code_F_n pq . Use Clang or Python's AST function library to parse source_code_F_n pq And generate an abstract syntax tree, identify the data type nodes (Clang's Type type node, Python's ast.Constant type node), exception type nodes (Clang's EnumConstantDecl type node) and tensor variable nodes (Clang's VarDecl node). If it is a data type node or an exception type node, go to 6.2.3.4.3.1.2.1.2.2; if it is a tensor variable node, go to 6.2.3.4.3.1.2.1.2.3.

[0288] 6.2.3.4.3.1.2.1.2.2 Use cosine similarity to find the most similar data type or exception type in the data type set or exception type set of the target deep learning framework (the data type set is usually located in the type.h file of the target deep learning framework, and the exception type set is located in the error.h file of the target deep learning framework), and parse the found type value into the corresponding data type node or exception type node, replacing source_code_F_n pq The original node in the image is mapped to the target framework after completion, and then go to 6.2.3.4.3.1.3;

[0289] 6.2.3.4.3.1.2.1.2.3 According to the tensor definition file of the target framework (usually located in Tensor.h of the target framework), make corresponding modifications to the tensor variable nodes, such as changing the calling method and initialization method of the tensor or updating related property settings, including device location, tensor shape, memory layout, gradient properties and data pointers. After completion, the above nodes are mapped to the target framework, and go to 6.2.3.4.3.1.3;

[0290] 6.2.3.4.3.1.2.2 According to the operation sequence seq_F_n pq The provided modification position key-value pairs (modification position, code line number) and seq_F_n pq The provided modification code key-value pair (source framework operator function code line, target framework operator function code line) is used to replace the F_n in the source framework operator function code line. p The relevant code is added to the target frame operator function code line to obtain the modified target frame operator function code line, and the modified target frame operator function code line is inserted at the modified position, the method is:

[0291] 6.2.3.4.3.1.2.2.1 Read seq_F_n pq The source framework operator function code line source_code_F_n in the provided modification code key-value pair pq ; Read seq_F_n pq The target framework operator function code line target_code_F_n in the provided modification code key-value pair pq ;

[0292] 6.2.3.4.3.1.2.2.2 Parsing source_code_F_n pq . Use Clang or Python's ast function library to parse source_code_F_n pq In the abstract syntax tree format (called F_n pq _source_AST), traverse F_n pq _source_AST, identification and F_n p Nodes with the same name and their parent nodes.

[0293] 6.2.3.4.3.1.2.2.3 Parse and modify target_code_F_n pq , add F_n pq Related functions. Use Clang or Python's ast function library to parse target_code_F_n pq In the abstract syntax tree format (called F_n pq _target_AST), traverse F_n pq _target_AST, using variable name similarity matching (i.e., Levenshtein distance), find F_n pq The node in _target_AST that is most similar to the parent node identified in 6.2.3.4.3.1.2.2.2, and F_n p Node inserted into F_n pqUnder the parent node of the node with the highest similarity in _target_AST. Complete F_n p Align related functions to get the modified target_code_F_n pq , go to 6.2.3.4.3.1.3;

[0294] 6.2.3.4.3.1.3 Let r=r+1. If r>R, then we get F_n pq After alignment, go to 6.2.3.4.3.2 for the target deep learning framework function code snippet; if r≤R, go to 6.2.3.4.3.1.2;

[0295] 6.2.3.4.3.2 F_n pq The aligned target deep learning framework function code snippet imports the target framework image classification model and aligns the operator parameters. The method is:

[0296] 6.2.3.4.3.2.1F_n pq After alignment, the target deep learning framework function code snippet will have at most two segments (one segment is F_n pq The aligned target deep learning framework Python function code snippet; the other is F_n pq Aligned target deep learning framework C function code snippet), if the programming language type of the function code is Python, go to 6.2.3.4.3.2.2; if the programming language type of the function code is C, go to 6.2.3.4.3.2.3;

[0297] 6.2.3.4.3.2.2 F_n pq The aligned target deep learning framework Python function code snippet is encapsulated into a python function and imported into the target deep learning framework image classification model. The method is:

[0298] 6.2.3.4.3.2.2.1 F_n pq Aligned target deep learning framework Python function code snippet and F_n pq The relevant function code compares the context list in the snippet (i.e. the sub-functions in the same file being called, the functions, namespaces and header files to which the function code snippet belongs), and generates a list containing F_n pq The Python file of the aligned target deep learning framework Python function code snippet (the steps of building a Python file are: create a new Python file, write the header file, the function to which the function code belongs, and the sub-function under the same file into the Python file, and the function to which the function code belongs contains F_n pqThe aligned target deep learning framework Python function code snippet, with the prefix "modified" added to the function name of the original function code), the sub-function under the same file, and the above content is written to the file).

[0299] 6.2.3.4.3.2.2.2 Use Python's import function (import statement) to import the Python file generated in 6.2.3.4.3.2.2.1 into the target framework image classification model and convert n p The operator name is changed to the function name of the function to which the function code belongs.

[0300] 6.2.3.4.3.2.2.3 Determine F_n pq Does the target F_n exist in the aligned target deep learning framework function code snippet? pq Aligned target deep learning framework C function code snippet. If there is F_n pq If the aligned target deep learning framework C function code snippet exists, go to 6.2.3.4.3.2.3; if it does not exist, go to 6.2.3.4.4;

[0301] 6.2.3.4.3.2.3 F_n pq The aligned target deep learning framework C function code snippet is encapsulated into a C file, and the C file is built into a dynamic link library (.so or .dll) through the target framework CMakeList template uploaded by the user. The dynamic link library is imported into the target framework image classification model. The method is:

[0302] 6.2.3.4.3.2.3.1 F_n pq Aligned target deep learning framework C function code snippet and F_n pq The relevant function code compares the context list in the snippet (i.e. the sub-functions in the same file being called, the functions, namespaces and header files to which the function code snippet belongs), and generates a list containing F_n pq The C file of the aligned target deep learning framework C function code fragment (the steps of generating the C file are: creating a new C file, writing the header file, namespace declaration, the function to which the function code belongs, and the sub-function under the same file into the C file, and the function to which the function code belongs contains F_n pq The aligned target deep learning framework C function code snippet is added with the prefix "modified" to the function name of the original function code).

[0303] 6.2.3.4.3.2.3.2 Read the compilation content in the CMakeList template, modify the first parameter of the add_library function, specify it as a compilable C file, and obtain the modified CMakeList template;

[0304] 6.2.3.4.3.2.3.3 Use the operating system instruction cmake to compile the CMakeList template modified in 6.2.3.4.3.2.3.2 to obtain the dynamic link library of the function code;

[0305] 6.2.3.4.3.2.3.4 Use the operating system instruction LD_PRELOAD or LoadLibrary to link the dynamic link library of 6.2.3.4.3.2.3.3;

[0306] 6.2.3.4.4 Let q = q + 1. If q > Q, complete n p Frame mapping, get the node n after cross-frame mapping p ', n p 'Put it in NN'; go to 6.3; if q≤Q, go to 6.2.3.4.2;

[0307] 6.2.4 Construct an equivalent target framework function expression based on the operator function call sequence provided in SS. The method is:

[0308] 6.2.4.1n p is an element of SO, ordseq_n p The corresponding ordseq_so is found in step 2.2.5.4 i . According to the n provided in SS p The operator function call sequence ordseq_n p , process ordseq_n in order p For each element in , reconstruct the equivalent functional code in the target framework. The method is:

[0309] 6.2.4.1.1 Initialize s = 1;

[0310] 6.2.4.1.2 Determine ordseq_n ps If the operation type is a target frame operator, go to 6.2.4.1.2.1; if the operation type is a source operator parameter, go to 6.2.4.1.2.2; if the operation type is a basic arithmetic operator, go to 6.2.4.1.2.3.

[0311] 6.2.4.1.2.1 Get ordseq_n ps Operator name (calling method, such as torch.conv2d), go to 6.2.4.1.3;

[0312] 6.2.4.1.2.2 Get ordseq_n ps Parameter value (if there is no parameter value, get the default parameter value), go to 6.2.4.1.3;

[0313] 6.2.4.1.2.3 Get ordseq_n ps For the corresponding operators in the target framework (e.g., in PaddlePaddle, addition corresponds to paddle.add, subtraction corresponds to paddle.divide, and multiplication corresponds to paddle.multiply), go to 6.2.4.1.3.

[0314] 6.2.4.1.3s Let s=s+1, if s>S, we get n p If the equivalent functional code is reconstructed in the target framework, go to 6.2.4.2; if s≤S, go to 6.2.4.1.2;

[0315] 6.2.4.2 n p The equivalent functional code in the target framework is reconstructed to parse into a syntax abstract tree (called ordseq_n p _AST), ordseq_n p _AST root node as the node n after cross-frame mapping p ', n p 'Put it to NN'.

[0316] 6.3 Let p = p + 1. If p>P, complete the cross-frame mapping of NN and obtain NN' (including all the frame mapped nodes n p '), go to 6.4; if p≤P, go to 6.2;

[0317] 6.4 Replace NN in the source AST with NN' to form the grammatical abstract tree of the target framework image classification model (referred to as target AST). The method is:

[0318] 6.4.1 Initialize p = 1;

[0319] 6.4.2 According to the CC provided n p The position in the source AST, corresponding to n in NN' p 'Replace n p ;

[0320] 6.4.3 Let p = p + 1. If p>P, NN in the source AST has been replaced by NN', go to 6.4.4; otherwise, go to 6.4.2;

[0321] 6.4.4 Use the to_source function in Python's astor library to convert the target AST into the target framework image classification model;

[0322] 6.5 Send the target framework image classification model to the image classification model weight loading module.

[0323] In the seventh step, the image classification model weight loading module receives the target framework image classification model from the cross-frame mapping module, and receives the source deep learning framework official model weights input by the user. Taking PyTorch as an example, open the PyTorch model Github library. Find the required model in the torchvision / models directory (model library directory), such as ResNet-50. On the introduction page of the ResNet-50 model, find the weight download link (that is, the URL of the pre-trained weights in the resnet.py file is usually specified in the function definition. For example, in the definition of ResNet-50, model_urls = {'resnet50':'https: / / download.pytorch.org / models / resnet50-19c8e357.pth',...}). Use the load function of the target framework image classification model to load the source deep learning framework official model weights, obtain the image classification inference model, and send the image classification inference model to the image classification module. The method is:

[0324] 7.1 Input the official model weights of the source deep learning framework into the state_dict function of the deep learning framework to obtain the official model weights in numpy format;

[0325] 7.2 Use the load function of the target framework image classification model to load the official model weights in numpy format into the target framework image classification model to obtain the image classification inference model;

[0326] 7.3 Send the image classification inference model to the image classification module.

[0327] In the eighth step, the image classification module receives the image classification inference model from the image classification model weight loading module, receives the image to be classified uploaded by the user, and outputs the classification result of the image. The method is:

[0328] The image classification module forward propagates to obtain the prediction result: the image to be classified uploaded by the user is input into the image classification inference model (the image is input in the form of RGB), and the image classification inference model outputs the classification result of the image to be classified.

[0329] In order to verify the effect of the present invention on the image classification task in the cross-frame model conversion scenario, a comparative experiment between the present invention and ONNX and PaConvert in the background technology was carried out on a server equipped with a 64-bit CentOS 7.9 operating system and a 52-core CPU (Intel(R) Xeon(R) Platinum 8358), four A100 GPUs and 394GRAM (ONNX is the image classification method based on cross-frame model conversion in "ONNX: Open Neural Network Exchange for Framework Interoperability" published by Bai X et al. at NurIPS19, and PaConvert is the image classification method based on cross-frame model conversion open sourced by Baidu on Github). The kernel version of the ContOS 7.9 operating system is 3.10.0, the software environment is PyTorch2.0.1, PaddlePaddle2.5.2.post117, python3.10 and CMake3.15, and the main coding languages ​​are Python and C++. The experimental process is as follows:

[0330] 1. Select 18 PyTorch image classification models and obtain the official weights of these 18 image classification models from the PyTorch model library.

[0331] 2. Use the image classification system based on cross-framework model conversion and operator difference alignment, ONNX, and PaConvert built in the first step of the present invention to convert the image classification model in the source framework (PyTorch) into the image classification model of the target framework (PaddlePaddle).

[0332] 3. Convert the official weights of PyTorch’s image classification model to Numpy format.

[0333] 4. Load the intelligent model weights in Numpy format into the image classification model of the target framework to obtain the image classification inference model of the target framework.

[0334] 5. Use the ImageNet dataset (from the public website HuggingFace, available at: https: / / huggingface.co / datasets / ILSVRC / imagenet-1k) (including test cases and correct classification of the test cases) as the test set, input each test case (i.e., the image to be classified) in the test set into the target framework image classification inference model, obtain the test results, compare them with the standard classification, and determine whether the test case passes.

[0335] The final performance evaluation indicators are: Precision, Recall, and F1 Score. The test results are shown in Tables 1 and 2.

[0336] Table 1 shows the performance of the present invention, ONNX, and PaConvert in terms of model support. Compared with ONNX and PaConvert, the present invention supports more image classification models for conversion (supports all image classification models to be tested);

[0337] Table 2 shows the performance of the target framework image classification inference model of the present invention, ONNX, and PaConvert on the image classification task. Table 2 summarizes 6 types of models based on the basic architecture of 18 inference models: AlexNet (including alexnet), VGG (including vgg11, vgg13, vgg16, vgg19), DenseNet (including densenet121, densenet161, densenet169, densenet201), ShuffleNet (including shufflenetv2_0_5, shufflenetv2_1_0, shufflenetv2_1_5, shufflenetv2_2_0), ResNet (including resnet18, resnet34, resnet50, resnet101), Inception3 (inception3). For the model types without operator differences (ResNet, Inception3, DenseNet, ShuffleNet), the present invention has better performance than ONNX and PaConvert. For models without operator differences (AlexNet, VGG), especially for ResNet and Inception3 model types, the classification accuracy of the target framework image classification inference model of the present invention is improved by up to about 4.5% (F1 of ResNet), and the stability of the classification accuracy is improved by 69.1% (measured by the standard deviation of the evaluation index). In addition, for models without operator differences (AlexNet, VGG), the performance of the present invention is on par with ONNX and PaConvert. In summary, the present invention is superior to the prior art in both performance and stability.

[0338] Table 1. Support of the present invention, ONNX, and PaConvert in model conversion

[0339]

[0340] Table 2. Performance of target framework image classification models of the present invention, ONNX, and PaConvert

[0341]

Claims

1. An image classification method based on cross-frame model conversion and operator difference alignment, characterized in that The following steps are involved: The first step is to build an image classification system based on cross-frame model conversion and operator difference alignment. The system consists of an operator difference analysis module, an operator difference mapping module, a model syntax parsing module, an operator analysis module, a cross-frame mapping module, an image classification model weight loading module, and an image classification module. The operator difference analysis module is an operator difference extraction tool, which is connected to the operator difference mapping module. It compares and analyzes the source deep learning framework operator list SO and the target deep learning framework operator list TO input by the user, extracts the differences in the operator interface and operator function code of the operators corresponding to SO and TO, and obtains the operator name, operator interface mapping relationship, operator difference type, operator function code comparison fragment and operator function call sequence; the operator name is a string type data, which stores the source deep learning framework operator name; the operator interface mapping relationship is a dictionary type data, including the operator name mapping relationship key-value pair (source deep learning framework operator name, target deep learning framework operator name), parameter mapping relationship key-value pair (key storage source deep learning framework The key-value pair is a tuple consisting of a key and a value; the operator difference type is a string type data, including three types, namely "operator function signature and usage difference", "operator behavior difference" and "operator missing difference"; the operator function code comparison fragment is a hierarchical dictionary, and the root level contains a key-value pair (the associated source deep learning framework parameter in the function code, the modification sequence); each element in the operator function call sequence is an operation, corresponding to one of the target framework operator, the source framework operator parameter, and the basic operator; the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence are sent to the operator difference mapping module; The operator difference mapping module is connected to the operator difference analysis module, the model syntax parsing module, and the cross-frame mapping module, and receives the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment, and operator function call sequence from the operator difference analysis module; maps the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment, and operator function call sequence into an operator mapping table SS; and sends the operator mapping table SS to the model syntax parsing module and the cross-frame mapping module; The model syntax parsing module is connected to the operator difference mapping module, the operator analysis module, the cross-frame mapping module, and the image classification model weight loading module, receives the source frame image classification model uploaded by the user, receives the operator mapping table SS from the operator difference mapping module, obtains the operator name and the source frame image classification model uploaded by the user from SS, and parses the source frame image classification model uploaded by the user into a source abstract syntax tree, i.e., a source AST; The source AST represents the components of the image classification model in an abstract way that is independent of the framework. The model syntax parsing module identifies and marks the nodes related to the operator in the source AST, and filters out the framework-related node set NN from the source AST through the framework root directory name, sends the NN and the source AST to the operator analysis module, and sends the NN to the cross-frame mapping module. The operator analysis module is connected to the model syntax parsing module and the cross-frame mapping module, receives the frame-related node set NN and the source AST from the model syntax parsing module, analyzes each node in NN, and obtains the context information CC of each node in NN in the source AST. The context information includes the operator name, operator parameter list, node type, node position, i.e., the code line where the node is located, and pointers to the parent node and child node. Each element in the operator parameter list includes the parameter name, parameter value, and parameter default value. The node type is ast.Call or ast.attribute, and ast is a standard library of Python. CC and the source AST are sent to the cross-frame mapping module. The cross-frame mapping module is connected to the operator difference mapping module, the model syntax parsing module, the operator analysis module, and the image classification model weight loading module. It receives the operator mapping table SS from the operator difference mapping module, receives the framework-related node set NN from the model syntax parsing module, receives the operator context information CC and the source AST from the operator analysis module, and receives the target framework CMakeList template uploaded by the user. It performs cross-frame mapping on each node in NN, and performs different difference alignment operations according to the operator difference type of each node in NN corresponding to the operator in SS. During the alignment process, it completes the operator interface mapping, the target framework operator Python function code alignment, and the target framework operator C function code alignment. Finally, the mapped frame-related node set NN' is obtained, and according to the node position in the operator context CC, the nodes in NN in the source AST are replaced with the corresponding nodes in NN' to obtain the abstract syntax tree of the target frame, i.e., the target AST. The target AST is reparsed into the target frame image classification model, and the target frame image classification model is sent to the image classification model weight loading module; The image classification model weight loading module is connected to the cross-frame mapping module and the image classification module, receives the official model weights of the source deep learning framework uploaded by the user, receives the target framework image classification model from the cross-frame mapping module, loads the official model weights of the source deep learning framework into the target framework image classification model, obtains the image classification inference model, and sends the image classification inference model to the image classification module; The image classification model module is connected to the image classification model weight loading module, receives the image to be classified uploaded by the user, receives the image classification inference model from the image classification model weight loading module, inputs the image to be classified uploaded by the user into the image classification inference model, and obtains the final classification result of the image to be classified; In the second step, the operator difference analysis module compares and analyzes the operator list SO of the source deep learning framework and the operator list TO of the target deep learning framework, and sends the analysis results, i.e., the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence, to the operator difference mapping module; the method is: 2.1 Operator difference analysis module receives the operator list SO of the source deep learning framework and the operator list TO of the target deep learning framework input by the user, SO = {so1, so2, ..., so i ,…,so I }, so i is the i-th operator in SO, I is the total number of operators in the source deep learning framework, 1≤i≤I; TO={to1,to2,…,to j ,…,to J }, to j is the jth operator in TO, J is the total number of operators in the target deep learning framework, 1≤j≤J; 2.2 Traverse SO, find the matching operator of each operator to be analyzed in SO in TO, and analyze the function codes of the two matching operators to obtain the function code comparison fragments of the two matching operators; the method is: 2.2.1 Construct an empty list NAME_TO and put to1, to2, …, to j ,…,to J Put it in NAME_TO; 2.2.2 Initialize i=1; 2.2.3 For the i-th operator so in SO i , will so i and NAME_TO are input to the get_close_matches function in Python's difflib library; the get_close_matches function is based on so i The operator name in NAME_TO is searched for an operator name that matches the operator name in NAME_TO, and a list containing all the operator names in NAME_TO that match the operator name in NAME_TO is obtained. i List of approximate matching operator names LL i , if LL i Not empty, execute 2.2.3.1; if LL i If it is empty, execute 2.2.3.2; 2.2.3.1 If LL is found in TO i The first element of finds the corresponding target frame operator in TO, let to j , the j for i Matching operator, go to 2.2.4; 2.2.3.2 Explanation: TO does not exist with so i Match the target frame operator, go to 2.2.5; 2.2.4 Analysis of so i and to j The function code is obtained i and to j Function code comparison snippet, the method is: 2.2.4.1 Generate so i and to j The interface mapping relationship is as follows: 2.2.4.1.1 Record so i and to j Operator name, formed so i and to j Operator name mapping key-value pairs (so i The operator name, to j ) 2.2.4.1.2 Initialize so i Parameter Mapping List SL i and the failed mapping parameter list fo i Is empty; traverse so i Operator parameter list param_so i ,param_so i = {param_so i1 ,param_so i2 ,…,param_so ik ,…,param_so iK }, param_so ik param_so i The kth parameter, K is the parameter in param_so i The total number of parameters, 1≤k≤K; the operator parameter name param_so ik As a keyword, in to j The parameter list param_to j Search in param_to j = {param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL }, L is param_to j The total number of parameters in, 1≤l≤L; param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL with param_so ik Perform name matching in to j Find the value corresponding to param_so ik The corresponding operator parameters are obtained to form so i Parameter Mapping List SL i and so i Failed to map parameter list fo i , SL i ={sl i1 ,sl i2 ,…,sl it ,…,sl iT }, sl it For SL i The tth so i and to j Parameter mapping key-value pairs, T is so i and to j The total number of parameter mapping key-value pairs; FO i ={fo i1 ,fo i2 ,…,fo im ,…,fo iM }, for im For FO i The mth mapping failure parameter in , M is FO i The total number of all failed parameters in; 2.2.4.1.3 Build the operator interface mapping relationship dictionary. The operator interface mapping relationship dictionary contains two key-value pairs, one for the so formed in step 2.2.4.1.1 i and to j The operator name mapping key-value pair and the parameter mapping list key-value pair ("parameter mapping list", SL i ), the dictionary is the operator interface mapping relationship; 2.2.4.2 Traversing FO i , follow up i1 ,fo i2 ,…,fo im ,…,fo iM Related operator implementation codes, obtain operator function code comparison snippets, the method is: 2.2.4.2.1 Initialize m = 1; 2.2.4.2.2 Use Python debugger PDB and GNU debugger GDB to run the operator i and to j : Use PDB to debug the code module written in Python and get the fo im Compare the relevant Python function code snippets; use GDB to debug the underlying code written in C / C++ and get the fo im Related C function code comparison snippets; merge fo im Related Python function code comparison snippets and fo im Compare the relevant C function code snippets to get the complete build fo im Related functional code comparison snippets, complete build fo im The relevant function code comparison snippet contains two key-value pairs: the associated parameter key-value pair ("associated parameter", fo im parameter name), operation sequence key-value pairs ("operation sequence", fo im Related Python function code comparison snippets and fo im Related C function code comparison snippets formed by combining the dictionary list); fo im Related Python function code comparison snippets and fo im The relevant C function code comparison snippet function code list key-value pairs are merged and defined as fo im The operation sequence seq_fo im , seq_fo im ={seq_fo im1 ,seq_fo im2 ,…,seq_fo imr ,…,seq_fo imR }, seq_fo imr For im The rth function code comparison fragment key-value pair, R is seq_fo imr fo im The total number of key-value pairs of the relevant function code comparison fragment; 2.2.4.2.3 Let m = m = +1; if m>M, change fo i1 ,fo i2 ,…,fo im ,…,fo iM The relevant functional code is compared and merged to obtain FO i For the relevant function code comparison fragment, go to 2.2.6; if m≤M, go to 2.2.4.2.2.1; 2.2.5 At this time, TO does not exist with so i Matching target frame operator, analysis so i The core calculation function kernelfunction is obtained i The operator function call sequence is: 2.2.5.1 Using GDB to debug so i , extract the function call stack of the operator program heap in GDB, and use the framework-related keywords, i.e., impl, functor, and kernel, to query the function call stack of the operator program heap in GDB. i Perform suffix matching on the function calls in the function call stack to filter out so i The kernel function in the function call stack; 2.2.5.2 Use the function syntax analysis method to analyze the core computing function kernel function, and form the operation sequence of the kernel function according to the actual execution order of the control flow; the method is: 2.2.5.2.1 Use Clang to generate an abstract syntax tree for the kernel function and obtain the AST of the kernel function; 2.2.5.2.2 Analyze the AST of the kernel function, process the return statement of the kernel function, and record the code line where the return statement is located; 2.2.5.2.3 Initialize an empty operation sequence list corresponding to the kernel function, which is used to record all relevant operations in the analysis process, including the target framework operator, the source framework operator parameters, the sub-functions called in the same file, and the arithmetic operators; 2.2.5.2.4 Use the syntax expression analysis method to analyze the return statement code of the kernel function and obtain the syntax expression of the return statement; The returnStmt node corresponding to the return statement in the kernel function's AST contains a return value expression in a tree structure. The local variables, namely clang's VarDecl node, the basic operators, namely clang's UnaryOperator node, clang's BinaryOperator node, clang's ConditionalOperator node, clang's CompoundAssignOperator node, the operator parameters, namely kernel function's parameters and clang's ParmVarDecl node, and the function call, namely clang's CallExpr node, are analyzed and identified. 2.2.5.2.5 Use the depth-first search method to traverse the syntax expression of the return statement in 2.2.5.2.4 to obtain the operation sequence of the kernel function; 2.2.5.3 Use the local variable analysis method to analyze all local variables, namely VarDecl nodes, in the kernel function's operation sequence; trace back the assignment statements of local variables in the kernel function, obtain the syntax expression of the local variables and replace the syntax expression with the local variables in the operation sequence to obtain the kernel function operation sequence that does not contain local variables; the method is: 2.2.5.3.1 Analyze the assignment statement of the local variable, i.e., the AssignExpr node, where the local variable is on the left side of the assignment statement and a syntax expression is on the right side of the assignment statement; use the syntax expression analysis method described in 2.2.5.2.4 to analyze the right side of the assignment statement, including the local variables, basic operators, operator parameters, and function calls therein; use the depth-first search method to construct the operation sequence of the AssignExpr node syntax expression, and replace the VarDecl node currently being analyzed in the operation sequence of the kernel function; 2.2.5.3.2 If there is no VarDecl node in the operation sequence of kernel function, complete the local variable analysis in the return statement to obtain the kernel function operation sequence without local variables, and go to 2.2.5.4; if there are still other VarDecl nodes in the operation sequence of kernel function, let the local variables currently being analyzed be other VarDecl nodes, and go to 2.2.5.3.1; 2.2.5.4 Use the function call analysis method to further analyze all function calls in the operation sequence of the kernel function, namely, CallExpr nodes, obtain the name of the function call of the CallExpr node, the parameter list in the function call, and suffix match the function call name with the target framework operator name. If the function call of the CallExpr node successfully matches the target framework operator name, the function call of the CallExpr node is replaced with the target framework operator; if the function call of the CallExpr node does not successfully match the target framework operator name, the function call code is deeply analyzed to form the operation sequence of the CallExpr node, and the CallExpr node in the operation sequence of the kernel function is replaced to obtain so i The operator function call sequence ordseq_so i , ordseq_so i ={ordseq_so i1 ,ordseq_so i2 ,…,ordseq_so is ,…,ordseq_so iS }, ordseq_so is is ordseq_so i The sth operation, S is ordseq_so i Total number of operations in, go to 2.2.6; 2.2.6 Let i=i+1; if i>I, go to 2.3; if i≤I, go to 2.2.3; 2.3 will so i The operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment or operator function call sequence are sent to the operator difference mapping module; In the third step, the operator difference mapping module receives the operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence from the operator difference analysis module, traverses the source deep learning framework operator list SO, and processes so1, so2, ..., so according to different operator difference types. i ,…,so I The operator name, operator difference type, operator interface mapping relationship, operator function code comparison fragment and operator function call sequence are mapped into an operator mapping table SS, and the SS is sent to the model syntax parsing module and the cross-frame mapping module; the method is: 3.1 Initialize the operator mapping table SS to be empty, and initialize i=1; 3.2 According to so i The operator difference type is handled differently: if the operator difference type is "operator function signature and usage difference", jump to 3.2.1; if the operator difference type is "operator behavior difference", jump to 3.2.2; if the operator difference type is "operator missing difference", jump to 3.2.3; 3.2.1 Construct a hierarchical dictionary to record three key-value pairs: operator name key-value pair ("operator name", so i operator name), operator difference type key-value pair ("operator difference type", so i operator difference type), operator interface mapping key-value pairs ("operator interface mapping", so i operator interface mapping relationship); add the hierarchical dictionary to SS and go to 3.3; 3.2.2 Construct a hierarchical dictionary to record four key-value pairs: operator name key-value pair ("operator name", so i operator name), operator difference type key-value pair ("operator difference type", so i operator difference type), operator interface mapping key-value pairs ("operator interface mapping", so i operator interface mapping relationship), operator function code comparison fragment key-value pairs ("operator function code comparison fragment", so i ); Add the hierarchical dictionary to SS and go to 3.3; 3.2.3 Build a hierarchical dictionary to record three key-value pairs: operator name key-value pair ("operator name", so i operator name), operator difference type key-value pair ("operator difference type", so i operator difference type), operator function call sequence key-value pairs ("operator function call sequence", so i operator function call sequence); add the hierarchical dictionary to SS and go to 3.3; 3.3 Let i=i+1; if i>I, it means that SO is processed and SS is obtained, go to 3.4; if i≤I, go to 3.2; 3.4 Send SS to the model syntax parsing module and cross-frame mapping module; In the fourth step, the model syntax parsing module receives the source frame image classification model uploaded by the user, and receives the operator difference mapping table SS from the operator difference mapping module; converts the source frame image classification model into AST, referred to as source AST, represents the various components of the source frame image classification model in a frame-independent manner, identifies and marks the nodes related to the operator, uses the root directory name of the frame, filters out the frame-related node set NN, and sends NN to the operator analysis module and the cross-frame mapping module; the method is: 4.1 Use the parse function in the Python standard library ast to convert the source framework image classification model code into the source AST; 4.2 Use the visit function of the source AST to recursively traverse the source AST nodes, identify each Call type or Attribute type node in the source AST, filter out the framework-related nodes, and obtain the framework-related node set NN containing all marked nodes, NN = {n1, n2, ..., n p ,…,n P },n p is the pth frame-related node of NN, i.e., operator, n p It has three attributes: name, which is the operator name; args, which is the operator parameter; and class, which is the node type corresponding to the operator in the source AST. 1≤p≤P, where P is the number of framework-related nodes in the NN. 4.3 Send NN and source AST to the operator analysis module and cross-frame mapping module; In the fifth step, the operator analysis module receives the frame-related node set NN and the source AST from the model syntax parsing module, parses the context information CC of the P frame-related nodes in NN in the source AST, and sends CC to the cross-frame mapping module. The method is: 5.1 Let p = 1; initialize the context information CC in the source AST to be empty, and initialize the node context information list CC p is empty; 5.2 Get the nth frame-related node n in NN p The upper and lower information of the source AST includes: operator name, operator parameter list, node type, node position, and pointers to parent nodes and child nodes. The method is: 5.2.1 Use the visit function of the source AST to traverse the source AST and record n p A list of parent nodes and child nodes; 5.2.2 Access n p Attribute name, get n p The name of the operator; 5.2.3 Access n p The attribute args, get n p Parameter list of 5.2.4 Access n p The attribute __class__ of p Node type; 5.2.5 Using astor.to_source function to obtain n p The code lines of the node in the source framework image classification model; 5.3 replace 5.2 with n p The operator name, operator parameter list, node type, node position, and pointers to parent nodes and child nodes are added to CC p In the p Join CC; 5.4 Let p = p + 1. If p > P, go to 5.

5. If p ≤ P, go to 5.

2. 5.5 Send CC to the cross-frame mapping module; In the sixth step, the cross-frame mapping module receives SS from the operator difference mapping module, receives NN from the model syntax parsing module, receives CC and source AST from the operator analysis node module, receives the target framework CMakeList template uploaded by the user, traverses the source AST, and performs cross-frame mapping for n1, n2, ..., n in NN. p ,…,n P Combine CC and SS to align n1,n2,…,n p ,…,n P The operator differences in the cross-deep learning framework mapping process are used to obtain the framework-related node set NN' after cross-frame mapping, and NN' is used to replace NN in the source AST to obtain the grammatical abstract syntax tree of the target framework, referred to as the target AST; the target AST is parsed into the target framework image classification model, and the target framework image classification model is sent to the image classification model weight loading module; the method is: 6.1 Let p = 1; initialize the frame-related node set NN' after cross-frame mapping to be empty; 6.2 According to n p The operator difference type in the operator difference mapping table SS is used to perform cross-frame mapping and align the operator differences in the cross-frame mapping process to obtain the cross-frame mapping node n. p '; The method is: 6.2.1 If the operator difference type is "Operator function signature and usage differences", go to 6.2.2; if the operator difference type is "Operator behavior differences", go to 6.2.3; if the operator difference type is "Operator missing differences", go to 6.2.4; 6.2.2 If the operator difference type is "Operator function signature and usage differences", complete n p Interface mapping, get n after cross-frame mapping p ';The method is: 6.2.2.1 Read n in SS p Operator interface mapping relationship, modify n according to the operator name in the mapping relationship p Name, complete n p Operator name mapping; 6.2.2.2 Access n p The args attribute is n p Parameter list, let n p The parameter list is param_n p , param_n p = {param_n p1 ,param_n p2 ,…,param_n po ,…,param_n pO }, param_n po param_n p The oth parameter of p The total number of parameters actually used, 1≤o≤O; 6.2.2.3 Access n p If the __class__ attribute is "attribute", go to 6.3; if the attribute is "Call", go to 6.2.2.4; 6.2.2.4 Using the operator parameter mapping method, according to n in SS p Parameter mapping list SL of interface mapping relationship i param_n p Mapping, get n p ', n p 'Put it to NN'; the parameter mapping method is: 6.2.2.4.1 Initialize o = 1; 6.2.2.4.2 For the oth param_n po , extract n from SS p Parameter mapping list SL in the interface mapping relationship i , find SL i to param_n po As a key-value pair, param_n po Replace with the corresponding value in the key-value pair; 6.2.2.4.3 Let o = o + 1. If o > O, then n is completed. p Interface mapping, obtain node n after cross-frame mapping p ', go to 6.2.2.5; if o≤O, go to 6.2.2.4.2; 6.2.2.5 n p 'Put to NN', go to 6.3; 6.2.3 For n p Perform operator interface mapping and operator parameter function alignment as follows: 6.2.3.1 Read n p Operator interface mapping relationship, modify n according to the operator name in the mapping relationship p Name, complete n p Operator name mapping; 6.2.3.2 Access n p args attribute, let n p The parameter list is param_n p ; 6.2.3.3 Using the operator parameter mapping method of 6.2.2.4, according to n in SS p Parameter mapping list SL of interface mapping relationship i param_n p After the mapping is completed, the type n p Only some parameters are mapped successfully, and some parameters fail to be mapped; 6.2.3.4 Collect the failed mapping parameters in 6.2.3.3 and get F_n p , F_n p ={F_n p1 ,F_n p2 ,…,F_n pq ,…,F_n pq }, F_n pq n p The qth failed mapping parameter of , where Q is the source frame operator n p Total number of failed mapping parameters, n p Corresponding to an element in SO, let the element be so i , F_n p So i Failed to map parameter list fo i A subset of p , using n in S p Operator function code comparison snippet, traversing FO i , align F_n p Related function codes, get the node n after cross-frame mapping p ', the method is: 6.2.3.4.1 Initialize q = 1; 6.2.3.4.2 Find FO i Chinese and Japanese F_n pq The key-value pair as the key. If it exists, the value of the key-value pair is F_n pq For the relevant function code comparison snippet, go to 6.2.3.4.3; if it does not exist, go to 6.2.3.4.4; 6.2.3.4.3 Traversing F_n pq Compare the operation sequence in the relevant function code snippet, modify the relevant function code snippet of the target deep learning framework, and complete the F_n of the target framework. pq Alignment, method: 6.2.3.4.3.1 Traversing F_n pq The operation sequence seq_F_n pq , F_n pq It is FO i Elements of F_n pq Corresponding to one of the fo in 2.2.4.1.2.5 im , seq_F_n pq Corresponding to a seq_fo in 2.2.4.2.2.3 im , seq_F_n pq is seq_F_n p elements, according to seq_F_n pq Each operation in modifies the target framework function code fragment to obtain F_n pq Aligned target deep learning framework function code snippet; 6.2.3.4.3.2F_n pq The aligned target deep learning framework function code snippet has at most two sections: one is F_n pq The aligned target deep learning framework Python function code snippet, and the other is F_n pq Aligned target deep learning framework C function code snippet; F_n pq The target deep learning framework function code snippet after alignment, import the target framework image classification model, align operator parameters, the method is: 6.2.3.4.3.2.1 F_n pq The aligned target deep learning framework Python function code snippet is encapsulated into a python function and imported into the target deep learning framework image classification model; the method is: 6.2.3.4.3.2.1.1 F_n pq Aligned target deep learning framework Python function code snippet and F_n pq The relevant function code is compared with the context list in the fragment to generate a list containing F_n pq The aligned Python files of the target deep learning framework Python function code snippets; 6.2.3.4.3.2.1.2 Use the python import function to import the Python file generated in 6.2.3.4.3.2.1.1 into the target framework image classification model and convert n p The operator name is changed to the function name of the function to which the function code belongs; 6.2.3.4.3.2.2 Determining F_n pq Does F_n exist in the aligned target deep learning framework function code snippet? pq Aligned target deep learning framework C function code snippet, if there is F_n pq If the aligned target deep learning framework C function code snippet exists, go to 6.2.3.4.3.2.3; if it does not exist, go to 6.2.3.4.4; 6.2.3.4.3.2.3 F_n pq The aligned target deep learning framework C function code snippets are encapsulated into C files, and the C files are built into dynamic link libraries through the target framework CMakeList template uploaded by the user; the dynamic link library is imported into the target framework image classification model as follows: 6.2.3.4.3.2.3.1 F_n pq Aligned target deep learning framework C function code snippet and F_n pq The relevant function code is compared with the context list in the fragment to generate a list containing F_n pq The C file of the aligned target deep learning framework C function code snippet; 6.2.3.4.3.2.3.2 Read the compilation content in the CMakeList template, modify the first parameter of the add_library function, specify it as a compilable C file, and obtain the modified CMakeList template; 6.2.3.4.3.2.3.3 Use the operating system command cmake to compile the modified CMakeList template to obtain the dynamic link library of the function code; 6.2.3.4.3.2.3.4 Use the operating system instruction LD_PRELOAD or LoadLibrary to link the dynamic link library; 6.2.3.4.4 Let q = q + 1. If q > Q, complete n p Frame mapping, get the node n after cross-frame mapping p ', n p 'Put it in NN'; go to 6.3; if q≤Q, go to 6.2.3.4.2; 6.2.4 Construct an equivalent target framework function expression based on the operator function call sequence provided in SS, the method is: 6.2.4.1 According to n provided in SS p The operator function call sequence ordseq_n p , process ordseq_n in order p For each element in ordseq_n, the equivalent function code is reconstructed in the target framework. p The method has been defined in 2.2.5.4.4 as: 6.2.4.1.1 Initialize s = 1; 6.2.4.1.2 Determine ordseq_n ps If the operation type is a target frame operator, go to 6.2.4.1.2.1; if the operation type is a source operator parameter, go to 6.2.4.1.2.2; if the operation type is a basic arithmetic operator, go to 6.2.4.1.2.3; the method is: 6.2.4.1.2.1 Get ordseq_n ps Operator name, go to 6.2.4.1.3; 6.2.4.1.2.2 Get ordseq_n ps If you want to know the parameter value or parameter default value, go to 6.2.4.1.3; 6.2.4.1.2.3 Get ordseq_n ps For the corresponding operators in the target framework, go to 6.2.4.1.3; 6.2.4.1.3 Let s = s + 1. If s > S, we get n p If the equivalent functional code is reconstructed in the target framework, go to 6.2.4.2; if s≤S, go to 6.2.4.1.2; 6.2.4.2 n p The equivalent functional code in the target framework is reconstructed and parsed into a syntax abstract tree called ordseq_n p _AST, ordseq_n p _AST root node as the node after cross-frame mapping n p ', n p 'Put to NN'; 6.3 Let p = p + 1. If p > P, complete the cross-frame mapping of NN to obtain NN', and go to 6.4; if p ≤ P, go to 6.2; 6.4 Replace NN in the source AST with NN' to form a grammatical abstract tree of the target framework image classification model, referred to as the target AST, and convert the target AST into the target framework image classification model; 6.5 Send the target framework image classification model to the image classification model weight loading module; In the seventh step, the image classification model weight loading module receives the target framework image classification model from the cross-frame mapping module, and receives the source deep learning framework official model weights input by the user, uses the load function of the target framework image classification model to load the source deep learning framework official model weights, obtains the image classification inference model, and sends the image classification inference model to the image classification module. The method is: 7.1 Input the official model weights of the source deep learning framework into the state_dict function of the deep learning framework to obtain the official model weights in numpy format; 7.2 Use the load function of the target framework image classification model to load the official model weights in numpy format into the target framework image classification model to obtain the image classification inference model; 7.3 Send the image classification inference model to the image classification module; In the eighth step, the image classification module receives the image classification inference model from the image classification model weight loading module, receives the image to be classified uploaded by the user, and outputs the classification result of the image; the method is: The image classification module forward propagates to obtain the prediction result: the image to be classified uploaded by the user is input into the image classification inference model in the form of RGB, and the image classification inference model outputs the classification result of the image to be classified.

2. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that The modification sequence in the operator function code comparison fragment is of array type, and the modification operation in the modification sequence is a sub-dictionary, which includes three fields: programming language type, function code fragment context information, and operation list; the programming language type is of string type; the function code fragment context information is of dictionary type; the operation list includes multiple specific modification operations, each of which is of dictionary type; the function code fragment context information includes three key-value pairs: sub-function call key-value pair ("sub-function call", sub-function call list), header file key-value pair ("header file", header file code), namespace key-value pair ("namespace", namespace of the source framework operator function code); the operation list includes multiple operations, and each operation includes 2 key-value pairs: modification position key-value pair ("modification position", code line number in the function where the function code is located), modification code key-value pair (function code fragment of the source framework, function code fragment of the target framework).

3. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that The SS is a hierarchical dictionary, and the root level is a key-value pair list. In each key-value pair in the key-value pair list, the key is the operator name and the value is a dictionary. It contains two or three key-value pairs according to the operator difference type: when the operator difference type is "operator function signature and usage difference", the value contains the operator difference type, operator interface mapping relationship key-value pair ("operator interface mapping relationship", operator interface mapping relationship); when the operator difference type is: "operator behavior difference", the value contains the operator difference type, operator interface mapping relationship key-value pair, operator function code comparison fragment key-value pair ("operator function code comparison fragment", storage operator function code comparison fragment); when the operator difference type is "operator missing difference", the value contains the operator difference type key-value pair, operator interface mapping relationship key-value pair, operator function call sequence key-value pair ("operator function call sequence", operator function call sequence).

4. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that 2.2.4.1.2 As described in step param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL with param_so ik Perform name matching in to j Find the value corresponding to param_so ik The corresponding operator parameter method is: 2.2.4.1.2.1 Initialize k = 1; 2.2.4.1.2.2 Traversing to j The parameter list param_to j , param_to j1 ,param_to j2 ,…,param_to jl ,…,param_to jL with param_so ik To match, use: 2.2.4.1.2.2.1 Initialize l = 1; 2.2.4.1.2.2.2 If param_so ik and param_to jl The names are the same, or their prefixes or suffixes partially match, then param_so ik and param_to jl Matching is successful; if matching is successful, a key-value pair is formed (param_so ik , param_to jl ), go to 2.2.4.1.2.3; if that fails, go to 2.2.4.1.2.2.3; 2.2.4.1.2.2.3param_so ik and param_to jl If the match fails, set l=l+1; if l>L, go to 2.2.4.1.2.4; if l≤L, go to 2.2.4.1.2.2.2; 2.2.4.1.2.3 At this time param_so ik and param_to jl Matching is successful, (param_so ik , param_to jl ) Add parameter mapping list SL i , let k=k+1, if k>K, go to 2.2.4.1.2.5; if k≤K, go to 2.2.4.1.2.2; 2.2.4.1.2.4 Explanation of param_so at this time ik With the target frame operator to j If all parameters in the match fail, param_so ik Add param_so i Failed mapping parameter list FO i , go to 2.2.4.1.2.5; 2.2.4.1.2.5 Now complete so i All parameters match, forming so i Parameter Mapping List SL i and so i Failed to map parameter list fo i , SL i ={sl i1 ,sl i2 ,…,sl it ,…,sl iT }, sl it For SL i The tth so i and to j Parameter mapping key-value pairs, T is so i and to j The total number of parameter mapping key-value pairs; FO i ={fo i1 ,fo i2 ,…,fo im ,…,fo iM }, for im For FO i The mth mapping failure parameter in , M is FO i The total number of all failed parameters in .

5. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that Run the operator using PDB and GDB combined debugging as described in step 2.2.4.2.

2. i and to j , record and fo im The relevant code to run the method is: 2.2.4.2.2.1 Building fo im Related Python function code comparison snippets: First, use PDB to debug the operator so i and to j , set breakpoints on the __init__ and forward functions in PDB; then run so i When the execution reaches the breakpoint, it enters the debug mode; in the debug mode, PDB uses the "enter function" instruction and the "next line" instruction to track the so i Python function code, execute the code line by line, and record so i fo im Related function code; for each line so i fo im Related function code, run to j When the execution reaches the breakpoint, it enters the debug mode; in the debug mode, PDB uses the "enter function" instruction and the "next line" instruction to track to j Python function code, execute the code line by line, use Python's ast parse function to parse so i fo im The relevant lines of code and to j The currently debugged code line uses the tree isomorphism, i.e. the AHU algorithm, to determine so by matching the node labels and the order and connection relationship of their child nodes. i fo im The subtree structure of the relevant code lines is the same as to j The similarity of the subtree of the currently debugged code line in the local structure is obtained, and the similarity score between the two is obtained; if so i fo im The subtree structure of the relevant code lines is the same as to j If the subtree structure of the currently debugged code line matches, then build so i fo im The Python function code sends snippet key-value pairs (so i Function code lines, to j Function code line); if to j Does not exist and so i fo im Related lines of code similar to the code, also build so i fo im Related Python function code snippet key-value pairs (so i Function code line, empty); build fo im A list of related Python function code contexts, including the called sub-functions in the same file, the functions to which the function code snippet belongs, namespaces, and header files; build fo im Related Python function code comparison snippets, fo im The relevant Python function code comparison snippet uses a dictionary data type, which contains three key-value pairs: programming language type key-value pair ("programming language type", "Python"), context information key-value pair ("context information", fo im Related Python function code context list), function code list key-value pairs ("function code key-value pair list", so i fo im Related Python function code snippet key-value pair list); 2.2.4.2.2.2 Construction of fo im Related C function code comparison snippet: Using GDB to debug the operator so i and to j , extract the function call stack of the operator program heap in GDB, use the keyword list containing impl, functor, and kernel to perform suffix matching, that is, the function name ends with the keyword, filter out the core calculation function kernel function, and set a breakpoint for the kernel function; in debug mode, run so i When the execution reaches the breakpoint, GDB uses the "enter function" instruction and the "next line" instruction to track the so i C function code, execute the code line by line, and record so i fo im Related function code; for each line so i fo im Related function code, run to j When the execution reaches the breakpoint, it enters the debugging mode; in the debugging mode, GDB uses the "enter function" instruction and the "next line" instruction to track to j The C function code is parsed using Clang i fo im Parameter related code lines and to j The current debugged code line uses the tree isomorphism algorithm to determine the similarity of the two trees in local structure by matching the order and connection relationship of node labels and their child nodes, and obtain the similarity score between the two trees; if so i fo im The subtree structure of the relevant code lines is the same as to j If the subtree structure of the currently debugged code line matches, then build so i fo im C function code snippet key-value pairs (so i The C function code line, to j C function code line); if to j Does not exist and so i fo im Related lines of code similar to the code, also build so i fo im Related C function code snippet key-value pairs (so i C function code line, empty); build fo im A list of related C function code contexts, including the called sub-functions in the same file, the functions to which the function code snippet belongs, the namespace and header files; build fo im Related C function code comparison snippet, fo im The relevant C function code comparison snippet contains three key-value pairs: programming language type key-value pair ("programming language type", "C"), context information key-value pair ("context information", fo im Related C function code context list), function code list key-value pairs ("function code key-value pair list", so i fo im Related C function code snippet key-value pair list); 2.2.4.2.2.3 Merge fo im Related Python function code comparison snippets and fo im Compare the relevant C function code snippets to get the complete build fo im Related functional code comparison snippets, complete build fo im The relevant function code comparison snippet contains two key-value pairs: the associated parameter key-value pair ("associated parameter", fo im parameter name), operation sequence key-value pairs ("operation sequence", fo im Related Python function code comparison snippets and fo im Related C function code comparison snippets formed by combining the dictionary list); fo im Related Python function code comparison snippets and fo im The relevant C function code comparison snippet function code list key-value pairs are merged and defined as fo im The operation sequence seq_fo im , seq_fo im ={seq_fo im1 ,seq_fo im2 ,…,seq_fo imr ,…,seq_fo imR }, seq_fo imr For im The rth function code comparison fragment key-value pair, R is seq_fo imr fo im Related function code comparison fragment key-value pairs total); 2.2.4.2.2.4 Let m=m=+1; if m>M, change fo i1 ,fo i2 ,…,fo im ,…,fo iM The relevant functional code is compared and merged to obtain FO i Compare the relevant function code fragments and end; if m≤M, go to 2.2.4.2.2.

1.

6. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 5, characterized in that The subtree structure matching described in step 2.2.4.2.2 means that the similarity score is greater than 0.

8.

7. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that In step 2.2.5.4, the function call analysis method is used to further analyze all CallExpr nodes in the operation sequence of the kernel function, obtain the name of the function call of the CallExpr node, the parameter list in the function call, and suffix match the function call name with the target framework operator name. If the function call of the CallExpr node successfully matches the target framework operator name, the function call of the CallExpr node is replaced with the target framework operator; if the function call of the CallExpr node does not successfully match the target framework operator name, the function call code is deeply analyzed to form the operation sequence of the CallExpr node, and the CallExpr node in the operation sequence of the kernel function is replaced. The method to obtain the kernel function operation sequence without the CallExpr node is: 2.2.5.4.1 Get the function name and parameter list of each CallExpr node in the kernel function's operation sequence; 2.2.5.4.2 Match the relationship between each CallExpr node in the kernel function's operation sequence and the target framework operator; check whether the function name in the CallExpr node matches the operator name in the target framework by exact matching or suffix matching; if the function name and parameters match successfully, replace the CallExpr node with the operator of the target framework and go to 2.2.5.4.4; if not, further analyze the CallExpr node and go to 2.2.5.4.3; 2.2.5.4.3 Use the function syntax analysis method in step 2.2.5.2 to analyze the function call content corresponding to the CallExpr node, and form the operation sequence of the CallExpr node according to the actual execution order of the control flow; Adopt the local variable analysis method of step 2.2.5.3 to analyze all VarDecl nodes in the operation sequence of the CallExpr node, trace back the assignment process of the local variables, obtain the syntax expression of the local variable and replace the syntax expression with the VarDecl node in the operation sequence, and obtain the operation sequence of the CallExpr node that does not contain the VarDecl node; replace the currently analyzed CallExpr node in the operation sequence of the kernel function with the operation sequence of the CallExpr node that does not contain the VarDecl node; 2.2.5.4.4 Continue to traverse the CallExpr nodes in the kernel function's operation sequence; if there are any unanalyzed CallExpr nodes, go to 2.2.5.4.1; If it does not exist, get so i The operator function call sequence ordseq_so i , ordseq_so i ={ordseq_so i1 ,ordseq_so i2 ,…,ordseq_so is ,…,ordseq_so iS }, ordseq_so is is ordseq_so i The sth operation, S is ordseq_so i The total number of operations in the sequence.

8. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that In step 4.2, the visit function of the source AST is used to recursively traverse the source AST nodes, identify each Call type or Attribute type node in the source AST, filter out the framework-related nodes, and obtain the method of the related node set NN: 4.2.1 Start traversing from the root node of the source AST, and let the current node v = the root node of the source AST; 4.2.2 Determine the node type of v. If the node type of v is a Call type or an Attribute type, go to 4.2.2.

2. If the node type of v is neither a Call type nor an Attribute type, go to 4.2.2.

1. 4.2.2.1 does not mark v and goes to 4.2.2.3; 4.2.2.2 Get the operator name in v and traverse so in SS i The operator name of is matched with the operator name in v. If the match is successful, v is determined to be a framework-related node and recorded. The method is: 4.2.2.2.1 Initialize the relevant node set NN to be empty, and initialize i=1; 4.2.2.2.2 Get so from SS i The operator name of v is obtained by obtaining the value of v, and so i The operator name of is matched with the operator name of v; if so i If the operator name of and the operator name of v match successfully, the node is marked and v is placed in NN, and the process goes to 4.2.2.

3. If the match fails, no marking is performed and the process goes to 4.2.2.2.

3. 4.2.2.2.3 Let i = i + 1. If i > I, go to 4.2.2.

3. If i ≤ I, go to 4.2.2.2.

2. 4.2.2.3 Traverse the next node of the source AST. If the next node exists, go to 4.2.

2. If the next node does not exist, it means that the framework-related node set NN containing all marked nodes is obtained, NN = {n1, n2, ..., n p ,…,n P }, 1≤p≤P, P is the number of framework-related nodes in NN.

9. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that 6.2.3.4.3.1 Traverse F_n pq The operation sequence seq_F_n pq , according to seq_F_n pq The method for modifying the target framework function code snippet for each operation in is: 6.2.3.4.3.1.1 Initialize r = 1; 6.2.3.4.3.1.2 For seq_F_n pq The rth operation in seq_F_n pqr , if seq_F_n pqr The second element of the modified code key-value pair is empty, go to 6.2.3.4.3.1.2.1; if seq_F_n pqr The second element of the modified code key-value pair is non-empty, go to 6.2.3.4.3.1.2.2; 6.2.3.4.3.1.2.1 According to the operation sequence seq_F_n pq The modification position key-value pair (modification position, code line number) provided by seq_F_npq and the modification code key-value pair (source framework operator function code line, empty) are used to map the source framework operator function code snippet to the target framework operator function code snippet, and insert the target framework operator function code snippet at the modification position. The method is: 6.2.3.4.3.1.2.1.1 Read seq_F_n pq The source framework operator function code line source_code_F_n in the provided modification code key-value pair pq ; 6.2.3.4.3.1.2.1.2 Source frame operator function code line source_code_F_n pq Map to the target framework and get target_code_F_n pq , the method is: 6.2.3.4.3.1.2.1.2.1 Parsing source_code_F_n pq ; Use Clang or Python's AST function library to parse source_code_F_n pq Generate an abstract syntax tree and identify the data type nodes, exception type nodes, and tensor variable nodes. If it is a data type node or an exception type node, go to 6.2.3.4.3.1.2.1.2.2; if it is a tensor variable node, go to 6.2.3.4.3.1.2.1.2.3; 6.2.3.4.3.1.2.1.2.2 Use cosine similarity to find the most similar data type or exception type in the data type set or exception type set of the target framework, and parse the found type value into the corresponding data type node or exception type node, replacing source_code_F_n pq The original node in the image is mapped to the target framework after completion, and then go to 6.2.3.4.3.1.3; 6.2.3.4.3.1.2.1.2.3 Modify the tensor variable nodes according to the tensor definition file of the target framework. After completion, map the above nodes to the target framework and go to 6.2.3.4.3.1.3; 6.2.3.4.3.1.2.2 According to the operation sequence seq_F_n pq The provided modification position key-value pairs (modification position, code line number) and seq_F_n pq The provided modification code key-value pair (source framework operator function code line, target framework operator function code line) is used to replace the F_n in the source framework operator function code line. p The relevant code is added to the target frame operator function code line to obtain the modified target frame operator function code line, and the modified target frame operator function code line is inserted at the modified position, the method is: 6.2.3.4.3.1.2.2.1 Read seq_F_n pq The source framework operator function code line source_code_F_n in the provided modification code key-value pair pq ; Read seq_F_n pq The target framework operator function code line target_code_F_n in the provided modification code key-value pair pq ; 6.2.3.4.3.1.2.2.2 Parsing source_code_F_n pq ; Use Clang or Python's ast function library to parse source_code_F_n pq In abstract syntax tree format, called F_n pq _source_AST, traverse F_n pq _source_AST, identification and F_n p Nodes with the same name and their parent nodes; 6.2.3.4.3.1.2.2.3 Parse and modify target_code_F_n pq , add F_n p Related functions: Use Clang or Python's ast function library to parse target_code_F_n pq In abstract syntax tree format, called F_n pq _target_AST, traverse F_n pq _target_AST, using variable name similarity matching, find F_n pq The node in _target_AST that is most similar to the parent node identified in 6.2.3.4.3.1.2.2.2, and insert the F_np node into F_n pq Under the parent node of the node with the highest similarity in _target_AST; complete F_n p Align related functions to get the modified target_code_F_n pq , go to 6.2.3.4.3.1.3; 6.2.3.4.3.1.3 Let r=r+1. If r>R, then we get F_n p For the modified target deep learning framework function code snippet, go to 6.2.3.4.3.2; if r≤R, go to 6.2.3.4.3.1.

2.

10. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 9, characterized in that 6.2.3.4.3.1.2.1.2.1 The data type nodes include Clang's Type type nodes and Python's ast.Constant type nodes, the exception type nodes include Clang's EnumConstantDecl type nodes, and the tensor variable nodes include Clang's VarDecl nodes; The data type set described in step 6.2.3.4.3.1.2.1.2.2 is located in the type.h file of the target deep learning framework, and the exception type set is located in the error.h file of the target deep learning framework; The tensor definition file described in step 6.2.3.4.3.1.2.1.2.3 is located in the Tensor.h file of the target framework. Corresponding modifications are made to the tensor variable nodes, including changing the tensor calling method, initialization method, or updating related property settings, wherein the property settings include device location, tensor shape, memory layout, gradient properties, and data pointers.

11. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that The method for generating a Python file containing the modified target framework function code snippet described in step 6.2.3.4.3.2.2.1 is: creating a new Python file, writing the header file, the function to which the function code belongs, and the sub-functions under the same file into the Python file, wherein the function to which the function code belongs contains the modified target framework function code snippet, and the prefix "modified" is added to the function name to which the original function code belongs; using the Python import function described in step 6.2.3.4.3.2.2.2 refers to using the import statement; The method for generating a C file containing a modified target framework function code fragment described in step 6.2.3.4.3.2.3.1 is: create a new C file, write the header file, namespace declaration, the function to which the function code belongs, and the sub-functions under the same file into the C file, the function to which the function code belongs contains the modified target framework function code fragment, and add the prefix "modified" to the name of the function to which the original function code belongs.

12. The image classification method based on cross-frame model conversion and operator difference alignment as claimed in claim 1, characterized in that In step 6.4, the method of replacing NN in the source AST with NN' to form the target AST and converting the target AST into the target framework image classification model is: 6.4.1 Initialize p = 1; 6.4.2 According to the CC provided n p The position in the source AST, corresponding to n in NN' p 'Replace n p ; 6.4.3 Let p = p + 1. If p>P, NN in the source AST has been replaced by NN', go to 6.4.4; otherwise, go to 6.4.2; 6.4.4 Use the to_source function in Python's astor library to convert the target AST into the target framework image classification model.

Images