Differential testing method and system for deep learning framework based on code semantic consistency

Abstract

The present invention discloses a deep learning framework differential testing method based on code semantic consistency, which uses semantic analysis technology in the field of natural language processing, static code analysis technology in the field of program analysis, and fuzz testing and metamorphosis testing concepts in the field of software testing to automatically capture defects in the deep learning framework. The method is capable of generating semantically consistent codes corresponding to multiple deep learning frameworks, and using multiple data transformation operators to generate input data for the deep learning framework to efficiently discover defects in the deep learning framework.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and in particular relates to a deep learning framework testing method and an automated testing method. Background Art

[0002] In recent years, with the continuous development and increasing maturity of artificial intelligence technology, intelligent software systems represented by deep learning software systems have gradually been integrated into various industries and different fields, playing an important role in people's daily work and life. For example, deep learning software has provided a variety of services including image recognition, speech recognition, data analysis and intelligent recommendation in many fields such as transportation, medical care, and e-commerce. In the military industry, deep learning software also embodies multiple application values ​​and has multiple advantages. Deep learning software systems provide solutions to problems in different industries and provide scientific and technological impetus for economic development and social progress. However, in this process, the quality problems existing in deep learning software have gradually been exposed and attracted people's attention. Different from the traditional software development process, the implementation of deep learning software systems is based on data-driven and underlying deep learning frameworks and other supporting components, facing a more open environment, and is more prone to unpredictable defects and behaviors.

[0003] At present, there are many kinds of deep learning frameworks. The widely used ones include TensorFlow released by Google, PyTorch released by Facebook, Paddlepaddle released by Baidu and MindSpore released by Huawei. Other deep learning frameworks include frameworks such as MXNet, Caffe, CNTK, Jittor, etc. Among them, TensorFlow was developed by researchers and engineers from the Google team, providing users with the function of creating neural networks and computational models using flowcharts. Developers can use C++ or Python to use the rich functions provided by the TensorFlow library. PyTorch is a Python-based deep learning framework released by Facebook, which can help deep learning researchers build deep learning models in the environment of dynamic computational graphs. Specifically, PyTorch provides a simple and effective way for deep learning software system developers to quickly define neural networks and run them efficiently on GPUs. Developers can use it to implement various deep learning algorithms, such as convolutional neural networks, recurrent neural networks, variational autoencoders, etc. PaddlePaddle is a deep learning framework released by Baidu. PaddlePaddle is based on Baidu's many years of deep learning technology research and business applications. It integrates deep learning core training and reasoning framework, basic model library, end-to-end development kit, and rich tool components. It is China's first independently developed, fully functional, open source industrial-grade deep learning framework. MindSpore is a full-scenario deep learning framework released by Huawei, aiming to achieve three goals: easy development, efficient execution, and full-scenario coverage. Easy development is manifested in API friendliness and low debugging difficulty. Efficient execution includes computing efficiency, data preprocessing efficiency, and distributed training efficiency. Full-scenario means that the framework supports cloud, edge, and end-side scenarios at the same time. The deep learning framework is the underlying application component that supports the development of intelligent software at this stage. Deep learning software developers can implement various complex deep learning algorithms and models by calling the high-level application program interface (API) provided by the deep learning framework without paying attention to the specific implementation inside the computing framework. Therefore, the specific implementation of deep learning software needs to rely on various deep learning algorithms and interfaces provided by the deep learning framework. Once these deep learning libraries or interfaces have code implementation errors and other problems, the deep learning software implemented based on the deep learning framework may have some defects that are difficult to detect and difficult to predict.

[0004] Therefore, as the defects in intelligent software gradually attract people's attention, more and more researchers and developers have studied the quality assurance of intelligent software systems as a topic and have made some progress. However, most of these existing studies are focused on the application level. These studies are often able to fully test a certain intelligent software system, but they are still lacking in the quality inspection of its underlying components. The functional defects of existing intelligent software systems can also come from the defects in the deep learning framework used. Therefore, it is particularly necessary to test the deep learning framework. However, in reality, the main problems in realizing automated testing of deep learning frameworks include missing test predictions, insufficient test input data and test cases, and high test complexity. Summary of the invention

[0005] Purpose of the invention: In traditional software testing, the generation of test cases is relatively direct and the construction of test predictions is more clear, which can usually be completed based on the requirements specification or design document. However, for deep learning frameworks, it is very difficult to generate specific test cases directly from the design and requirements of highly abstract deep learning models. Secondly, due to the high complexity and context dependence of the output results of deep learning models, it is also very challenging to clearly define an expected output for each test case. In addition, deep learning models have data-driven characteristics, and the existing test input data cannot guarantee the breadth of deep learning framework testing, and lacks effective input data to trigger possible defects.

[0006] In view of the problems and shortcomings existing in the prior art, the present invention provides a deep learning framework differential testing method and system based on code semantic consistency to solve the problems of lack of high-quality test cases, lack of test input data and difficulty in constructing test predictions in current deep learning framework testing.

[0007] Technical solution: A differential testing method for deep learning frameworks based on code semantic consistency. First, semantic analysis technology in the field of natural language processing and static code analysis technology in the field of program analysis are used to complete cross-framework deep learning API matching and API parameter alignment to achieve semantically consistent code generation across frameworks. Then, based on the differential testing theory in the field of software testing, by running semantically consistent code, the calculation results and running status of corresponding codes in different frameworks are compared to capture defects in the deep learning framework. In order to enrich the diversity of input data, that is, the necessary input data for code running, the present invention will also use a variety of data transformation operators to mutate the data in the existing data set to further improve the test adequacy.

[0008] The input of this method is a seed code file for building a deep learning model implemented in Python, and the output is multiple code files that capture inconsistent calculation results and running status. The number of code files is determined by the number of deep learning frameworks tested, and the minimum number is 2.

[0009] The method consists of four parts:

[0010] (1) Semantically consistent equivalent identification: Equivalents refer to APIs that implement the same functions in different deep learning frameworks, as well as parameters with alignment in deep learning frameworks. APIs across frameworks can be matched one-to-one or one-to-many, that is, API APIs in deep learning framework A can be implemented in deep learning framework B. a The corresponding functions need to execute the API sequence in order Parameters with alignment relationships in deep learning frameworks mean that the same functions can be achieved by setting their values ​​in matching APIs. Code blocks that call different APIs are the basic components of deep learning model codes. Changing parameters in code blocks corresponds to different code semantics. Therefore, obtaining matching APIs and aligned parameters is the basis for subsequent steps.

[0011] (2) Seed test case abstraction: Seed test cases refer to code files that implement deep learning models. Seed test case abstraction refers to converting specific code files that implement deep learning models into abstract file formats that represent the implementation process of the deep learning model. The main content covered by this abstract file is the structural information of the implemented deep learning model and the neuron information of each layer. This information is closely related to the type, parameters, and calling order of the layer API called in the seed code file. This step is designed to simplify the process of generating semantically consistent cross-framework code, treating the abstract file as an intermediary in the cross-framework code conversion process, and converting the many-to-many matching problem into a one-to-many translation problem.

[0012] (3) Generation of new test cases: The abstract model file is split into an abstract template and an abstract layer block, wherein the abstract template determines the data type and dimension that the model can receive, as well as information such as the output format, corresponding to the input layer and output layer of the deep learning model, and the remaining layer information is stored in the abstract layer block set L. In order to improve the diversity of the generated test cases, the present invention implements a mutation operator on the abstract layer to generate more abstract layer blocks that can be adapted to the current abstract template. After the abstract layer blocks and templates are assembled in sequence to generate multiple abstract model files, the abstract files are translated into model implementation codes corresponding to different deep learning frameworks according to the equivalents and parameters with alignment relationships identified in step (1) as new test cases.

[0013] (4) Test case execution: The process of deep learning model is divided into two stages: the model definition stage and the model training and verification stage. In the model definition stage, possible defects are captured based on the inconsistency of the code execution state; in the model training and verification stage, possible defects are captured based on the numerical inconsistency of the calculation results. In addition, in order to improve the adequacy of the test, the image data transformation operator is used to scale, perspective, affine and add noise to the data in the existing image test set, and then input it to the model in the training and verification stage.

[0014] A deep learning framework differential testing system based on code semantic consistency, including the following modules:

[0015] (1) Semantically consistent equivalent identification module: Equivalent refers to APIs that implement the same function in different deep learning frameworks, as well as parameters with alignment in deep learning frameworks. The APIs across frameworks can be matched one-to-one or one-to-many, that is, API APIs in deep learning framework A can be implemented in deep learning framework B. a The corresponding functions need to execute the API sequence in order Parameters with alignment relationships in deep learning frameworks mean that the same functions can be achieved by setting their values ​​in matching APIs. Code blocks that call different APIs are the basic components of deep learning model codes. Changing parameters in code blocks corresponds to different code semantics. Therefore, obtaining matching APIs and aligned parameters is the basis for subsequent steps.

[0016] (2) Seed test case abstract module: Seed test cases refer to code files that implement deep learning models. Seed test case abstraction refers to converting specific code files that implement deep learning models into abstract file formats that represent the implementation process of the deep learning model. The main content covered by this abstract file is the structural information of the implemented deep learning model and the neuron information of each layer. This information is closely related to the type, parameters, and calling order of the layer API called in the seed code file. This step is designed to simplify the process of generating semantically consistent cross-framework code, treating the abstract file as an intermediary in the cross-framework code conversion process, and converting the many-to-many matching problem into a one-to-many translation problem.

[0017] (3) New test case generation module: The abstract model file is split into an abstract template and an abstract layer block, wherein the abstract template determines the data type and dimension that the model can receive, as well as information such as the output format, corresponding to the input layer and output layer of the deep learning model, and the remaining layer information is stored in the abstract layer block set L. In order to improve the diversity of the generated test cases, the present invention implements a mutation operator on the abstract layer to generate more abstract layer blocks that can be adapted to the current abstract template. After the abstract layer blocks and templates are assembled in sequence to generate multiple abstract model files, the abstract files are translated into model implementation codes corresponding to different deep learning frameworks as new test cases.

[0018] (4) Test case execution module: The process of deep learning model is divided into two stages: the model definition stage and the model training and verification stage. In the model definition stage, possible defects are captured based on the inconsistency of the code execution state; in the model training and verification stage, possible defects are captured based on the numerical inconsistency of the calculation results. In addition, in order to improve the adequacy of the test, the data transformation operator is used to mutate the data in the existing test set, and then input to the model in the training and verification stage.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the deep learning framework differential testing method based on code semantic consistency as described above is implemented.

[0020] A computer-readable storage medium stores a computer program for executing the deep learning framework differential testing method based on code semantic consistency as described above.

[0021] Beneficial effects: Compared with the prior art, the deep learning framework differential testing method based on code semantic consistency provided by the present invention has the following effects:

[0022] (1) Realize API matching and parameter alignment across deep learning frameworks.

[0023] (2) Based on the seed code file that implements the deep learning model, a new code file with consistent semantics that is compatible with multiple frameworks can be generated.

[0024] (3) Apply the theories of model testing and differential testing to capture defects in deep learning frameworks based on inconsistency in execution state and inconsistency in calculated values. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a general flow chart of an embodiment of the present invention;

[0026] Figure 2A schematic diagram illustrating the abstract process in key step (2);

[0027] Figure 3 The figure is a detailed flow chart of the key step (3). DETAILED DESCRIPTION

[0028] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0029] Differential testing across deep learning frameworks based on code semantic consistency is a new attempt. Different from previous deep learning framework testing methods, this method uses semantic analysis technology in the field of natural language processing, static code analysis technology in the field of program analysis, and differential testing theory in the field of software testing to solve the problems of lack of high-quality test cases, lack of test input data, and difficulty in building test predictions in current deep learning framework testing.

[0030] The differential testing method of deep learning framework based on code semantic consistency includes the following:

[0031] (1) Identification of semantically consistent equivalents

[0032] API semantic consistency means that different APIs in different deep learning frameworks are consistent or highly similar in function. There are two types of matching relationships between them: one-to-one or one-to-many. For example, API API P belonging to deep learning library A is a and APIP in Deep Learning Library B b Or API sequence Have a matching relationship. The semantic consistency of parameters means that the functional meaning, default value, data type, value range and impact effect of the parameters with semantic alignment relationship in the API pair with matching relationship are consistent or highly similar. In particular, when the semantically consistent parameter value changes within the allowed range, the impact on the performance or behavior of the deep learning model should be consistent in the APIs of different frameworks. This embodiment uses ONNX as the medium for matching between various frameworks, and indirectly completes the API matching across frameworks by realizing the direct mapping of operators in ONNX to various frameworks. The Open Neural Network Exchange (ONNX, Open Neural Network Exchange) format is a standard for representing deep learning models, so that different deep learning frameworks can use the same format to store deep learning models. Usually, developers use ONNX to store trained models for easy deployment on different frameworks, but the present invention focuses on code rather than deep learning models, that is, it is not to convert the model trained using framework A into ONNX format and then into the model trained using framework B, but to convert the model information in the code file using framework A into ONNX format and then generate a code file using framework B based on the ONNX file. For example, from the API level, the mapping transfer chain of PyTorch and TensorFlow is torch.nn.Dropout->ONNX, Dropout->tf.keras.layers.Dropout. Therefore, in this step, the present invention focuses on the API matching and parameter alignment from ONNX to each framework (the adapted deep learning frameworks are PyTorch, TensorFlow, Mindspore and PaddlePaddle).

[0033] In order to ensure the correctness and completeness of the recognition results, Figure 1As shown in (1), the APIs with semantic consistency and the implementation parameter alignment are identified from two perspectives. First, according to the official documents, the API name information and definition information are processed and matched respectively. In terms of API names, since the ONNX operator does not consider the dimension of the data, this embodiment first uses English word segmentation technology to remove the dimension information in the API names of the four frameworks (for example, 1d, 2d and 3d in PyTorch), and then converts all the names to lowercase, and then obtains the APIs whose processed API names in different frameworks are exactly the same as the ONNX operator names as equivalent APIs. After the above process is completed, this embodiment uses cosine similarity to obtain the similarity between the API names in different frameworks and the ONNX operator names, and APIs greater than 0.85 are regarded as equivalent APIs. If an equivalent API cannot be matched based on the API name information, an equivalent API match is performed based on the API definition information, that is, the basic definition of the API in the deep learning framework and the definition of the ONNX operator are used to calculate the cosine similarity, and the APIs greater than 0.6 are regarded as equivalent APIs. For parameter alignment and ONNX operators that have not yet completed matching, this embodiment adopts a matching method based on a model converter. A model converter is a tool or library for converting machine learning and deep learning models from one format to another, and often supports cross-framework model conversion. Existing converters between ONNX and other frameworks include onnx2tf, onnx2torch, x2paddle, and mindconverter. This embodiment uses the Joern tool to perform static program analysis on the above-mentioned model converter, generate a control flow graph and an abstract syntax tree, and extract the required equivalent API information from the control flow graph and the abstract syntax tree, that is, the ONNX operator specifically corresponds to an API in the framework.

[0034] Align parameters in the two parameter lists of one-to-one equivalent API pairs. First, align the parameters of APIP in framework A. a The parameter names are lowercase. In framework B, APIP b The lowercase parameter list is searched for parameters with the same name completion. a If a parameter with the same name cannot be found in APIP b In the parameter list of , we lock the candidate set with the same data type, and find the parameters with the most similar parameter names as the alignable parameters. First, we convert the parameter names into sentence vectors, and then use cosine similarity to calculate the similarity of the parameter names. If the highest similarity is less than 0.6, it is considered that there are no suitable parameters in the candidate set for alignment, and then we extract the alignable parameters and the values ​​representing the same situation from the static analysis results provided by the Joern tool. For one-to-many equivalent APIs, we analyze them according to the document information and manually align the parameters.

[0035] (2) Abstract seed test case

[0036] The ONNX (Open Neural Network Exchange) IR (Intermediate Representation) format is selected to store the model structure information represented by the specific deep learning model implementation code. The core of the ONNX IR structure includes the following parts:

[0037] 1) Graph: ONNX uses graphs to represent models. Each graph contains a series of nodes, which represent operations (such as convolution, pooling, etc.), and edges connecting these operations, representing data flow.

[0038] 2) Node: A node is the basic unit in a graph, representing an operation or calculation in a model. Each node can have input and output. Input is data from other nodes, and output can be sent to other nodes.

[0039] 3) Attributes: Attributes provide additional information about the behavior of a node. For example, a convolution node may have attributes about the kernel size, stride, padding, etc.

[0040] 4) Data type: ONNX supports multiple data types, including integers, floating-point numbers, strings, etc., to meet different data requirements and calculation accuracy requirements.

[0041] 5) Operators: Operators define the specific operations performed by nodes in the graph. ONNX provides a rich library of operators that covers most standard operations used in deep learning, such as addition, multiplication, LSTM, etc.

[0042] 6) Input and Output: The input / output vectors representing the graph, including the data shape.

[0043] From the above content, we can know that the abstract test case is to convert the information in the code file into graphs, nodes, tensors, attributes, data types, and operators in the ONNX file. In the abstract process, the above concepts are first matched with the code semantics, that is, the graph corresponds to the deep learning model that the code file ultimately wants to implement, the node corresponds to the layer in the deep learning model that the code file ultimately wants to implement, the operator corresponds to the layer API called in the model construction phase of the code file, the attribute corresponds to the parameter value in the layer API, the data type corresponds to the data type of the parameter in the layer API, and the input and output correspond to the input layer and output layer in the code file. Based on the official documentation of ONNX, find the code block corresponding to the ONNX operator from the seed code file, and complete the abstract transformation according to the syntax rules of ONNX as the basis for subsequent steps. The abstract process is as follows Figure 2 shown.

[0044] (3) Test case generation

[0045] like Figure 3 As shown, according to the syntax rules of ONNX and the semantics of building deep learning model code files, the abstract model file is split into abstract layers and abstract templates. The abstract layer refers to the layer represented by ONNX, that is, the node. The corresponding parameter configuration and the connection order of the layer are the basic data for subsequent mutation; the abstract template includes the input layer and output layer of the model, which specifies the most basic context and provides guidance information in the mutation process. The present invention first mutates the abstract layers in the order in which they appear in the seed model file, and then inserts them into the template in sequence to complete the assembly of the new model abstract file. Since a new model abstract file will be generated for each inserted layer, the assembly step will eventually produce a multi-layer and multi-branch model abstract file tree. The following two mutation operators are used to mutate the nodes:

[0046] 3) Operator replacement mutation: Calculate the similarity of ONNX operators based on definition and parameter similarity to replace an ONNX operator with another ONNX operator with similar functionality, and generate parameters for the new operator based on the existing layer information to make it conform to the existing context constraints. The definition similarity calculation process is to first convert the definition of the operator into a vector, and then use the cosine similarity calculation method to measure the definition similarity between two operators. The parameter similarity calculation process is to treat the operator parameter list as a word sequence, and then calculate the Levenshtein distance between the two operator parameter lists. The average value of the definition and parameter similarity is taken as the operator similarity.

[0047] 4) Parameter variation: Based on the constraints of the existing layer, the parameter value range given in the official documentation, and other information, the existing parameter value is changed to another legal value that conforms to the semantics.

[0048] Based on the API matching information and parameter alignment information obtained in step (1), the model abstract file tree is translated into code file trees corresponding to different frameworks. The nodes at the same position in the code file trees of different frameworks are translated from the same ONNX file, so they contain the same code semantics, which is the basis for subsequent differential testing. Based on this feature, two test predictions are designed, one is state consistency and the other is numerical consistency. State consistency targets the model definition stage in the code file. The formal representation of state consistency is:

[0049]

[0050] Given an input set G, when the code is implemented using a code file P using framework P code and code file T implemented using framework T codeThe semantics of are completely consistent. For any input x in the input set G, P code With T code The execution status of the tested frameworks should be consistent. The specific execution status includes successful execution, exception throwing and crash. In the model definition stage, if the execution status of the related code files of multiple tested frameworks is inconsistent, it means that there may be defects such as document errors and API implementation errors, which need to be paid special attention.

[0051] The formal representation of numerical consistency is:

[0052]

[0053] Given an input set G, when the code is implemented using a code file P using framework P code and code file T implemented using framework T code The semantics of P are completely consistent. For any input x in the input set G, code With T code The outputs between layers of the constructed model should be completely equal or have small differences. During the training and verification phase of the model, the outputs, loss values, and gradient values ​​between layers in the forward calculation process, loss value calculation process, and backward calculation process of the same model built using different frameworks should have small differences. If there are large differences, it indicates that the tested framework may have defects and needs to be paid special attention.

[0054] (4) Test case execution

[0055] Select rotation, flipping, scaling, cropping, translation, color transformation, noise injection, and blurring to transform the existing image test set as the data set required by the code file. Run all code files in the code file tree of each framework, compare the state and value of the nodes at the same position to see if they meet the state consistency and value consistency, and find out the nodes with differences. According to the generational relationship in the code file tree, the code block where the defect exists can be confirmed, and the defect location of the deep learning framework can be completed.

[0056] A deep learning framework differential testing system based on code semantic consistency, including the following modules:

[0057] (1) Semantically consistent equivalent identification module: Equivalent refers to APIs that implement the same function in different deep learning frameworks, as well as parameters with alignment in deep learning frameworks. The APIs across frameworks can be matched one-to-one or one-to-many, that is, API APIs in deep learning framework A can be implemented in deep learning framework B. a The corresponding functions need to execute the API sequence in order Parameters with alignment relationships in deep learning frameworks mean that the same functions can be achieved by setting their values ​​in matching APIs. Code blocks that call different APIs are the basic components of deep learning model codes. Changing parameters in code blocks corresponds to different code semantics. Therefore, obtaining matching APIs and aligned parameters is the basis for subsequent steps.

[0058] (2) Seed test case abstract module: Seed test cases refer to code files that implement deep learning models. Seed test case abstraction refers to converting specific code files that implement deep learning models into abstract file formats that represent the implementation process of the deep learning model. The main content covered by this abstract file is the structural information of the implemented deep learning model and the neuron information of each layer. This information is closely related to the type, parameters, and calling order of the layer API called in the seed code file. This step is designed to simplify the process of generating semantically consistent cross-framework code, treating the abstract file as an intermediary in the cross-framework code conversion process, and converting the many-to-many matching problem into a one-to-many translation problem.

[0059] (3) New test case generation module: The abstract model file is split into an abstract template and an abstract layer block, wherein the abstract template determines the data type and dimension that the model can receive, as well as information such as the output format, corresponding to the input layer and output layer of the deep learning model, and the remaining layer information is stored in the abstract layer block set L. In order to improve the diversity of the generated test cases, the present invention implements a mutation operator on the abstract layer to generate more abstract layer blocks that can be adapted to the current abstract template. After the abstract layer blocks and templates are assembled in sequence to generate multiple abstract model files, the abstract files are translated into model implementation codes corresponding to different deep learning frameworks as new test cases.

[0060] (4) Test case execution module: The process of deep learning model is divided into two stages: the model definition stage and the model training and verification stage. In the model definition stage, possible defects are captured based on the inconsistency of the code execution state; in the model training and verification stage, possible defects are captured based on the numerical inconsistency of the calculation results. In addition, in order to improve the adequacy of the test, the data transformation operator is used to mutate the data in the existing test set, and then input to the model in the training and verification stage.

[0061] The system implementation process and method are the same and will not be repeated here.

[0062] Obviously, those skilled in the art should understand that the various steps of the deep learning framework differential testing method based on code semantic consistency or the various modules of the deep learning framework differential testing system based on code semantic consistency of the above-mentioned embodiments of the present invention can be implemented by a general-purpose computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A differential testing method for a deep learning framework based on code semantic consistency, characterized in that: The methods include semantically consistent equivalent identification, seed test case abstraction, test case generation, and test case execution; Semantically consistent equivalent identification, extracting valid information from the deep learning framework's documentation, using static code analysis technology to extract control flow graphs and abstract syntax trees from deep learning model converters, and completing API matching and parameter alignment; Seed test case abstraction. A seed test case is a code file that implements a deep learning model. Seed test case abstraction refers to converting a specific code file that implements a deep learning model into an abstract file format, an ONNX file, that represents the implementation process of the deep learning model. The abstract file includes the structural information of the deep learning model and the neuron information of each layer. According to the syntax rules of ONNX, convert the semantics in the code into ONNX files; Test case generation: According to the syntax rules of ONNX and the semantics of building deep learning model code files, the abstract model file is split into abstract layers and abstract templates; Operators are replaced or parameters are mutated according to the order in which the abstract layers appear in the seed model file, and then inserted into the template in sequence to generate new model abstract files, which can eventually generate a model abstract file tree; Afterwards, the model abstract file tree is translated into code file trees corresponding to different frameworks; Test case execution: transform the existing image test set as the data set required by the code file; run all code files in the code file tree of each framework, compare the running status and calculation process of the nodes at the same position to see whether they meet the state consistency and value consistency, and find out the nodes with differences; Based on the generational relationship in the code file tree, the code block where the defect exists can be confirmed to complete the defect location of the deep learning framework.

2. The deep learning framework differential testing method based on code semantic consistency according to claim 1 is characterized in that: The input of the method is a seed code file for building a deep learning model implemented in Python, and the output is multiple code files that capture inconsistent calculation results and running status.

3. The deep learning framework differential testing method based on code semantic consistency according to claim 1 is characterized in that: In the semantically consistent equivalent identification, equivalents refer to APIs that implement the same functions in different deep learning frameworks, as well as parameters with alignment relationships in deep learning frameworks; APIs across frameworks are matched one-to-one or one-to-many, that is, API APIs in deep learning framework A are implemented in deep learning framework B. a The corresponding functions need to execute the API sequence in order Parameters with aligned relationships in the deep learning framework mean that the same functions can be achieved by setting their values ​​in APIs with matching relationships. Code blocks that call different APIs are the basic building blocks for implementing deep learning model codes. Changing parameters in code blocks corresponds to different code semantics. Therefore, obtaining APIs with matching relationships and parameters with aligned relationships is the basis for subsequent steps.

4. The deep learning framework differential testing method based on code semantic consistency according to claim 1 is characterized in that: In the test case generation, rotation, flipping, scaling, cropping, translation, color change, noise injection and blurring are selected to transform the existing image test set, and the abstract model file is split into abstract templates and abstract layer blocks, wherein the abstract template determines the data type and dimension that the model can receive, as well as the output format information, corresponding to the input layer and output layer of the deep learning model, and the remaining layer information is stored in the abstract layer block set L; a mutation operator is implemented on the abstract layer to generate more abstract layer blocks that can be adapted to the current abstract template; after the abstract layer blocks and templates are assembled in sequence to generate multiple abstract model files, the abstract files are translated into model implementation codes corresponding to different deep learning frameworks as new test cases.

5. The deep learning framework differential testing method based on code semantic consistency according to claim 1 is characterized in that: The compatible deep learning frameworks include TensorFlow, PyTorch, PaddlePaddle and Mindspore.

6. The deep learning framework differential testing method based on code semantic consistency according to claim 1 is characterized in that: In the test case generation, according to the syntax rules of ONNX and the semantics of building the deep learning model code file, the abstract model file is split into abstract layers and abstract templates; the abstract layer refers to the layer represented by ONNX, that is, the node, and the corresponding parameter configuration and the connection order of the layer are the basic data for subsequent mutation; the abstract template includes the input layer and the output layer of the model; first, the abstract layers are mutated in the order in which they appear in the seed model file, and then inserted into the template in sequence to generate a new model abstract file; since each layer inserted will generate a new model abstract file, a multi-layer and multi-fork model abstract file tree will eventually be generated; the following two mutation operators are used to mutate the nodes: 1) Operator replacement mutation: Replace the ONNX operator with another operator based on the definition and parameter similarity, and generate parameters that meet the context constraints for the new operator based on the layer information; the definition similarity calculation process is to first convert the operator definition into a vector, and then use the cosine similarity calculation method to measure the definition similarity between two operators; the parameter similarity calculation process is to regard the operator parameter list as a word sequence, and then calculate the Levenshtein distance between the two operator parameter lists; the average value of the definition and parameter similarity is taken as the operator similarity; 2) Parameter variation: Based on the layer constraints and parameter value range information, the parameter value is changed to another legal value that conforms to the semantics; Based on the obtained API matching information and parameter alignment information, the model abstract file tree is translated into code file trees corresponding to different frameworks; The nodes at the same position in the code file tree of different frameworks are translated from the same ONNX file, so they contain the same code semantics. Two test predictions are designed, one is state consistency and the other is value consistency. State consistency is aimed at the model definition stage in the code file. The formal expression of state consistency is: Given an input set G, when the code is implemented using a code file P using framework P code and code file T implemented using framework T code The semantics of P are completely consistent. For any input x in the input set G, code With T code The execution status of the tested frameworks should be consistent. The specific execution status includes successful execution, exception throwing and crash. In the model definition stage, if the execution status of the related code files of multiple tested frameworks is inconsistent, it means that there may be defects such as document errors and API implementation errors. The formal representation of numerical consistency is: Given an input set G, when the code is implemented using a code file P using framework P code and code file T implemented using framework T code The semantics of P are completely consistent. For any input x in the input set G, code With T code The inter-layer outputs of the constructed model should be exactly equal, or the difference should be less than the set value; during the training and verification phases of the model, the inter-layer outputs, loss values, and gradient values ​​in the forward calculation process, loss value calculation process, and backward calculation process of the same model built using different frameworks should differ by less than the set value, otherwise it indicates that the tested framework may have defects and needs to be paid special attention.

7. A deep learning framework differential testing system based on code semantic consistency, characterized in that: The method includes a semantically consistent equivalent body recognition module, a seed test case abstraction module, a test case generation module and a test case execution module; The semantically consistent equivalent recognition module extracts valid information from the deep learning framework's documentation, uses static code analysis technology to extract control flow graphs and abstract syntax trees from the deep learning model converter, and completes API matching and parameter alignment; Seed test case abstraction module. The seed test case is a code file that implements the deep learning model. Seed test case abstraction refers to converting the specific code file that implements the deep learning model into an abstract file form, ONNX file, that represents the implementation process of the deep learning model. The abstract file includes the structural information of the deep learning model and the neuron information of each layer. According to the syntax rules of ONNX, convert the semantics in the code into ONNX files; Test case generation module: splits the abstract model file into abstract layers and abstract templates according to the syntax rules of ONNX and the semantics of building deep learning model code files; Operators are replaced or parameters are mutated according to the order in which the abstract layers appear in the seed model file, and then inserted into the template in sequence to generate new model abstract files, which can eventually generate a model abstract file tree; Afterwards, the model abstract file tree is translated into code file trees corresponding to different frameworks; Test case execution module: transform the existing image test set as the data set required by the code file; run all code files in the code file tree of each framework, compare the running status and calculation process of the nodes at the same position to see whether they meet the state consistency and value consistency, and find out the nodes with differences; Based on the generational relationship in the code file tree, the code block where the defect exists can be confirmed to complete the defect location of the deep learning framework.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the deep learning framework differential testing method based on code semantic consistency as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the deep learning framework differential testing method based on code semantic consistency as described in any one of claims 1 to 7.

Images