Model stability test method and device, computer equipment and storage medium

Abstract

The invention relates to a model stability test method and device, computer equipment and a storage medium. The method comprises the steps that a current reasoning process is executed on a target model, the current reasoning process comprises multiple times of reasoning with the same test data, a current calculation node is obtained, the current calculation node is a node corresponding to an unstable model output tensor obtained by the current reasoning process, an input tensor of the current calculation node is obtained, and the input tensor of the current calculation node is obtained; and performing a next reasoning process according to the input tensor of the current computational node, the next reasoning process including multiple reasoning with the same test data, and when the input tensors of the current computational node in each reasoning result of the next reasoning process are consistent, determining that the current computational node is an unstable node. By adopting the method, the unstable operator can be quickly positioned, and the operator stability testing efficiency is improved.

Description

Model stability testing methods, apparatus, computer equipment, and storage media Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model stability testing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Model stability testing is used to verify whether the model's output results and performance metrics remain consistent and free from abnormal drift under different environments, small fluctuations in input data distribution, or long-term operation. By conducting model stability testing, we can prevent sudden changes in prediction results due to environmental differences or data fluctuations after deployment, thus avoiding impacts on the accuracy of the model's predictions.

[0003] Traditional techniques typically involve running the model on the target hardware and different framework versions, then manually comparing the output results. When the model's stability test fails, it is necessary to manually locate the unstable operator, which is inefficient. Summary of the Invention

[0004] Therefore, it is necessary to provide a model stability testing method, apparatus, computer equipment, and storage medium that can improve the efficiency of model stability testing in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a model stability testing method, comprising: executing a current inference process on a target model, the current inference process including multiple inferences with the same test data; obtaining a current computation node, the current computation node being the node corresponding to an unstable model output tensor obtained from the current inference process, the unstable model output tensor being a model output tensor with inconsistent inference results in each inference of the current inference process; obtaining the input tensor of the current computation node, and performing a next inference process based on the input tensor of the current computation node, the next inference process including multiple inferences with the same test data; determining the current computation node as an unstable node when the input tensor of the current computation node is consistent in each inference result of the next inference process; and determining the stability test result of the target model based on the unstable node.

[0006] In one embodiment, the unstable model output tensors obtained in the current inference process include multiple tensors. The model stability testing method further includes: storing the unstable model output tensors obtained in the current inference process into a preset queue; obtaining the current computing node, including: obtaining and deleting the current tensor from the preset queue; obtaining the computing node corresponding to the current tensor to obtain the current computing node; when the node type of the current computing node is in a preset whitelist and the number of variable inputs of the current computing node is one, adding the variable inputs to the preset queue; obtaining and deleting the next tensor from the preset queue, using the next tensor as the current tensor, and returning to the step of obtaining the computing node corresponding to the current tensor.

[0007] In one embodiment, obtaining the input tensor of the current computing node includes: obtaining the current input tensor of the current computing node when the node type of the current computing node is not in a preset whitelist and / or the number of variable inputs of the current computing node is not the same; the model stability testing method further includes: determining whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed; when the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed, skipping the current input tensor and obtaining the next input tensor of the current computing node; returning to the step of determining whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed; when the current input tensor is not a constant, and the current input tensor is not an input tensor of the target model, and the current input tensor is not a tensor that has been accessed, executing the step of proceeding to the next inference process based on the input tensor of the current computing node.

[0008] In one embodiment, performing the next inference process based on the input tensor of the current computing node includes: marking the input tensor of the current computing node as the model output tensor of the target model to adjust the output of the target model; and performing the next inference process using the adjusted model based on the input tensor of the current computing node. The model stability testing method further includes: obtaining the unstable model output tensor of the next inference process, wherein the unstable model output tensor of the next inference process is the model output tensor in which inconsistent inference results exist in each inference result of the next inference process; when the unstable model output tensor of the next inference process contains the marked current input tensor, adding the current input tensor to a preset queue; and when the unstable model output tensor of the next inference process does not contain the marked current input tensor, determining that the current input tensor has consistent results in each inference of the next inference process.

[0009] In one embodiment, before obtaining the next input tensor of the current computing node, the model stability test method further includes: determining whether all input tensors of the current computing node have been traversed; determining the current computing node as an unstable node when the input tensors of the current computing node are consistent in each inference result of the next inference process, including: if all input tensors of the current computing node have been traversed and the input tensors of the current computing node are consistent in each inference result of the next inference process, then the current computing node is determined to be an unstable node; obtaining the next input tensor of the current computing node includes: obtaining the next input tensor of the current computing node when there are input tensors of the current computing node that have not been traversed.

[0010] In one embodiment, the model stability testing method further includes: in response to a release request for the target model, extracting the model weights of the target model and storing them in a storage system; in response to a reproduction request for the target model, retrieving the model weights and model code of the target model from the storage system; converting the model code into an operator computation graph based on the model weights, and reproducing the target model based on the operator computation graph.

[0011] In one embodiment, the model stability testing method further includes: determining the original training model of the target model based on the model weights and model code; performing inference on the target model and the original training model of the target model using test data to obtain a first result and a second result respectively; calculating the value of a pre-configured accuracy index based on the first result and the second result respectively, and comparing them; determining the accuracy alignment result between the target model and the original training model of the target model based on the comparison result, so as to perform accuracy testing.

[0012] In one embodiment, the model stability testing method further includes: in response to a code merge request in the current code repository, extracting the code modification portion from the code merge request, wherein the current code repository is the repository where the target model's code resides; when the code modification portion contains information about the model deployment framework corresponding to the target model, performing model stability testing and accuracy testing on all models corresponding to the model deployment framework in the code repository; when the code modification portion contains common code of the target model, performing model stability testing and accuracy testing on each sub-model of the target model; when the code modification portion is code of a sub-model of the target model, performing model stability testing and accuracy testing on the sub-model.

[0013] Secondly, this application provides a model stability testing device, comprising: a current inference module for executing a current inference process on a target model, the current inference process including multiple inferences with the same test data; an acquisition module for acquiring a current computing node, the current computing node being the node corresponding to an unstable model output tensor obtained from the current inference process, the unstable model output tensor obtained from the current inference process being a model output tensor with inconsistent inference results in each inference of the current inference process; a next inference module for acquiring the input tensor of the current computing node and performing a next inference process based on the input tensor of the current computing node, the next inference process including multiple inferences with the same test data; a processing module for determining the current computing node as an unstable node when the input tensor of the current computing node is consistent in each inference result of the next inference process; and a determination module for determining the stability test result of the target model based on the unstable node.

[0014] In a third aspect, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the model stability testing method provided in any embodiment of the first aspect of this application.

[0015] In a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the model stability testing method provided in any embodiment of the first aspect of this application.

[0016] The aforementioned model stability testing method, apparatus, computer equipment, and storage medium first input the same test data to the target model and perform the current inference process to obtain the unstable model output tensor output by the current inference process. Then, it associates the corresponding computation nodes in the operator computation graph to obtain the input tensor of the current computation node. The current computation node is the node corresponding to the unstable model output tensor obtained by the current inference process. The next inference process is then performed on the input tensor of the current computation node. Each inference process includes multiple inference processes with the same test data. If all the input tensors of the current computation node are consistent in each inference of the next inference process, the current computation node is determined to be an unstable node, i.e., an unstable operator. This achieves rapid location of unstable operators. Compared with the manual location of unstable operators in traditional technologies, this application improves the efficiency of operator stability testing. Attached Figure Description

[0017] Figure 1 is a flowchart of a model stability testing method in some embodiments; Figure 2 is a flowchart of a model stability testing method in other embodiments; Figure 3 is a flowchart of an accuracy testing process in some embodiments; Figure 4 is a flowchart of an accuracy testing process for multiple GPUs (Graphics Processing Units) in some embodiments; Figure 5 is a flowchart of a code merging process in some embodiments; Figure 6 is a structural block diagram of a model stability testing device in some embodiments; Figure 7 is an internal structural diagram of a computer device in some embodiments. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] In a first aspect, this application provides a model stability testing method, as shown in Figure 1. Taking the application of this method to a terminal as an example, the method includes the following steps: Step S11, execute the current inference process on the target model. The current inference process includes multiple inferences with the same test data.

[0020] Among them, model stability testing refers to the testing process of verifying whether the core indicators such as output results, accuracy, and latency of the deployed or to-be-deployed model remain consistent and without abnormal drift under the same input, different environments, small data fluctuations or long-term operation scenarios, so as to ensure the availability and reliability of model services.

[0021] The target model is the model currently used to perform inference. It can be a deployment model such as an ONNX (Open Neural Network Exchange) model. A deployment model refers to a model that, after training and validation, is packaged into a service form that can run in production environments such as servers, the cloud, and edge devices. It can receive real or test inputs and output prediction results.

[0022] Test data refers to a set of samples, such as images, text, and numerical features, that meet the model input format and cover the core scenarios, and is used to verify the model's performance and stability.

[0023] The current inference process refers to multiple inference operations performed on the target model with the same test data as input. The core objective is to filter out model output tensors that produce inconsistent inference results, i.e., unstable model output tensors. Inference refers to the process of the model receiving input data after loading and outputting prediction results through internal calculations. "Same test data" means that the same test data is used in each inference operation of the current inference process.

[0024] In this application, multiple prediction calculations are repeatedly performed on a model already deployed to the production environment, and the consistency of the model output after multiple runs is observed to verify its stability. Specifically, the test data used in each inference iteration is the same as step S12, which involves obtaining the current computation node. The current computation node is the node corresponding to the unstable model output tensor obtained in the current inference process. The unstable model output tensor obtained in the current inference process refers to the model output tensor with inconsistent inference results in each inference iteration of the current inference process.

[0025] The inference result refers to the final result output by the model after receiving input data once.

[0026] The model output tensor refers to the tensor that the model outputs after completing one inference iteration. A tensor is the core data carrier for model computation; it is a multi-dimensional array that carries the numerical information of the input, intermediate computation results, and final output.

[0027] An unstable model output tensor refers to a situation in which the model output tensor corresponding to the same node obtained under different inference operations in the current inference process has inconsistent values, where the test data for each inference operation is the same.

[0028] Specifically, this application can collect the model output tensors of the target model in multiple inferences in the current inference process, classify and store them according to the number of inferences and tensor names, and compare the results of multiple inferences one by one for tensors with the same name, filter out the model output tensors whose values ​​are inconsistent with the results of other inferences, and record them as unstable model output tensors.

[0029] Step S13: Obtain the input tensor of the current computing node, and perform the next inference process based on the input tensor of the current computing node. The next inference process includes multiple inferences with the same test data.

[0030] Here, the current computation node is an operator in the operator computation graph of the target model. The operator computation graph of the target model is a graph that clearly describes the end-to-end computational logic of the model from input to output in the form of nodes and edges, where nodes represent operators and edges represent tensor transfers. Operators are the basic computational units in the computation graph and are the core components for the model to perform specific operations.

[0031] Input tensors refer to the input data received by a computing node during inference.

[0032] The next inference process involves multiple inference operations performed again based on the input tensor of the computation node located in the current inference process. The core purpose is to verify whether the input tensor of that computation node is stable, thereby determining whether the computation node is unstable. "Same test data" means that the test data for each inference operation in the next inference process is identical.

[0033] Specifically, this application can load the operator computation graph of the target model, locate the position of the unstable model output tensor in the computation graph, find the current computation node, and extract all the input tensors actually received by the current computation node during the inference process from the association relationship of the computation graph.

[0034] Step S14: When the input tensor of the current computation node is consistent in each inference result of the next inference process, the current computation node is determined to be an unstable node.

[0035] Unstable nodes refer to computational nodes where, in each inference operation of the inference process, the input test data is the same, the input tensor is the same, and there is no external interference, the output results are still inconsistent after multiple inferences.

[0036] This application can compare the inference results of each inference process in the next inference process to determine whether the value of the input tensor of the current computing node is consistent in each inference result. If they are consistent, it means that the input tensor has no fluctuation. The inconsistency problem originates from the node's own operation, and the node is determined to be an unstable node. If the output results are inconsistent, it is necessary to further investigate the upstream nodes of the input tensor.

[0037] Step S15: Determine the stability test results of the target model based on the unstable nodes.

[0038] Among them, the stability test results refer to the number and identification of unstable operators contained in the target model.

[0039] This application can combine the found unstable nodes to give a final judgment on the stability of the model. If no unstable nodes are found, the model is determined to be stable. If unstable nodes exist, the location or identifier, number and impact of the unstable nodes are output. Finally, based on the unstable operators, the conclusion of whether the model meets the standard and whether it needs to be optimized can be output to complete the stability test.

[0040] In one embodiment, the unstable model output tensor obtained in the current inference process includes multiple tensors. The model stability testing method further includes: storing the unstable model output tensors obtained in the current inference process into a preset queue, and obtaining the current computing node, including: obtaining and deleting the current tensor from the preset queue, obtaining the computing node corresponding to the current tensor, obtaining the current computing node, adding the variable input to the preset queue when the node type of the current computing node is in a preset whitelist and the number of variable inputs of the current computing node is one, obtaining and deleting the next tensor from the preset queue, using the next tensor as the current tensor, and returning to the step of obtaining the computing node corresponding to the current tensor.

[0041] The preset whitelist refers to a pre-defined set of node types that do not require in-depth investigation. These nodes have been verified to have very few instability issues and can be skipped from subsequent verification.

[0042] Variable input refers to non-constant input, which is the input tensor received by the computing node that may change with the data or environment during the inference process, as opposed to fixed constant input.

[0043] The preset queue is a cache queue used to store tensors to be investigated. Initially, unstable model output tensors are stored, and intermediate input tensors that meet the conditions can be added later.

[0044] The current tensor is a tensor to be checked taken from the preset queue. It may be the initial model output tensor or an input tensor added later in the intermediate process. When the first inference process is executed, the preset queue stores unstable model output tensors. As the method runs, other intermediate tensors may be added to the preset queue.

[0045] Specifically, this application can store the unstable model output tensor obtained from the current inference process into a preset queue, retrieve and delete a tensor from the head of the preset queue, and record it as the current tensor. Find the computation node that outputs the current tensor; determine whether the type of the computation node is in the preset whitelist and whether the number of its variable inputs is 1; if both of the above conditions are met, add the variable input of the computation node to the preset queue; retrieve the next tensor from the preset queue as the new current tensor, return to the step of retrieving the computation node corresponding to the current tensor, and repeat the execution.

[0046] In this application, each time a tensor is taken from the preset queue, it is determined whether there are any remaining unprocessed tensors in the preset queue. If there are, the next tensor is taken from the preset queue. If not, it means that all tensors in the preset queue have been processed, the process ends, and the unstable nodes found are output.

[0047] Additionally, each time a tensor is retrieved from the preset queue, it is simultaneously removed from the preset queue. When a found node is not an unstable node, its input tensor is added to the preset queue for further investigation until the final unstable node is found. Once an unstable node is found, its input is no longer added to the preset queue. This process continues until no tensors remain in the preset queue, at which point the process ends, and all unstable nodes obtained throughout the process are output.

[0048] The beneficial effect of this embodiment is that stable operators are filtered by whitelist and variable input, eliminating the need to traverse the entire tensor and nodes, avoiding redundant checks on meaningless nodes, and improving the efficiency of stability testing.

[0049] In one embodiment, obtaining the input tensor of the current computing node includes: obtaining the current input tensor of the current computing node when the node type of the current computing node is not in a preset whitelist and / or the number of variable inputs of the current computing node is not the same. The model stability testing method further includes: determining whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed; when the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed, skipping the current input tensor, obtaining the next input tensor of the current computing node, and returning to the step of determining whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed.

[0050] The input tensor of the current computing node includes at least one. When there are multiple input tensors, they need to be checked one by one. Specifically, the current input tensor is obtained first and checked. The current input tensor is one of the multiple input tensors of the computing node.

[0051] Specifically, this application first determines that the node type of the computation node corresponding to the current tensor is not in the preset whitelist, or the number of variable inputs is not 1. It then obtains the current input tensor of the current computation node and determines whether it is a constant, the original input tensor of the model, or a tensor that has been visited. If so, it skips the current input tensor of the current computation node, obtains the next input tensor of the current computation node, and returns to the step of determining whether the current input tensor is a constant, the input tensor of the target model, or a tensor that has been visited.

[0052] If the current input tensor is a non-constant, not the original input of the model, and has not been accessed, then the next inference step based on the input tensor of the current computing node will be executed.

[0053] The beneficial effect of this embodiment is that by setting a second skip logic, constants, the original input of the model, and tensors that have been visited are excluded, meaningless reasoning verification is reduced, and the efficiency of stability testing is further improved.

[0054] In one embodiment, the next inference process based on the input tensor of the current computing node includes: when the current input tensor is not a constant, is not the input tensor of the target model, and is not a tensor that has been visited, marking the current input tensor as the model output tensor of the target model to adjust the output of the target model, and using the adjusted model to perform the next inference process based on the current input tensor. The model stability testing method further includes: obtaining the unstable model output tensor of the next inference process, wherein the unstable model output tensor of the next inference process is the model output tensor with inconsistent inference results in each inference result of the next inference process; when the unstable model output tensor of the next inference process contains the marked current input tensor, adding the current input tensor to a preset queue; when the unstable model output tensor of the next inference process does not contain the marked current input tensor, determining that the current input tensor has consistent results in each inference of the next inference process.

[0055] Specifically, by modifying the output list of the target model, the labeled input tensor is added to the output list of the target model. The updated model is then deployed to the inference engine, and the inference engine outputs the model based on the updated output list. When inference is performed multiple times, the target model will eventually output the labeled input tensor. This is equivalent to adding the intermediate output of the model to the final output list of the model, thereby changing the final output of the model.

[0056] The marked current input tensor refers to the current input tensor that is marked as the model output tensor. When determining the stability of the current input tensor, this application first modifies the target model's output, marking the current input tensor as the target model's output tensor. This is equivalent to modifying the target model's output list. The model will then output based on the new output list, and the final output tensor will include this marked current input tensor. Further, unstable output tensors are filtered from the model's output tensors. It is then determined whether the unstable output tensors include the input tensor of the currently marked computation node. If they do, the input tensor of this current computation node is unstable and is added to a pre-defined queue. If they do not, the current input tensor is stable, and the next input tensor is checked.

[0057] The advantage of this embodiment is that it eliminates the need for repeated model conversion and allows for the direct acquisition of the stability results of the input tensors of the target computation node by adjusting the final output list of the target model.

[0058] In one embodiment, before obtaining the next input tensor of the current computing node, the model stability test method further includes: determining whether all input tensors of the current computing node have been traversed; if the input tensors of the current computing node are consistent in each inference result of the next inference process, the current computing node is determined to be an unstable node, including: if all input tensors of the current computing node have been traversed and the input tensors of the current computing node are consistent in each inference result of the next inference process, then the current computing node is determined to be an unstable node; obtaining the next input tensor of the current computing node includes: obtaining the next input tensor of the current computing node when there are input tensors of the current computing node that have not been traversed.

[0059] Before obtaining the next input tensor of the current computing node, all input tensors of the computing node must be traversed. Only when all input tensors have been traversed and the results of all input tensors in each inference in the current inference process are consistent, is the current settlement node determined to be an unstable node. If there are still input tensors that have not been traversed, the next input tensor is obtained for further investigation.

[0060] Please refer to Figure 2, which is a flowchart of a model stability testing method in one embodiment. In Figure 2, the same test data is input to the target model to execute the current inference process. The current inference process includes multiple inference operations. The inference results of each inference operation in the current inference process are obtained. Unstable model output tensors are selected from these inference results. Furthermore, the current computation node is obtained based on the unstable model output tensors of the current process. The current computation node is the node corresponding to the unstable model output tensor obtained by the current inference process.

[0061] Further stability tests are performed on each input tensor of the current computing node to verify whether each input tensor of the current computing node is stable. If it is, the current computing node is an unstable node; otherwise, the upstream nodes of the current computing node are investigated.

[0062] Specifically, obtain the current input tensor of the current computing node, determine whether the current input tensor of the current computing node is a constant, an input tensor of the target model, or a tensor that has been visited, and if the current input tensor is a constant, an input tensor of the target model, or a tensor that has been visited, skip the current input tensor, obtain the next input tensor of the current computing node, and return to the step of determining whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been visited.

[0063] If the current input tensor is not constant, is not the input tensor of the target model, and is not a previously visited tensor, then the current input tensor is marked as the output tensor of the target model to adjust the output of the target model. The adjusted model is then used to proceed to the next inference process based on the current input tensor. The unstable model output tensor for the next inference process is obtained. The unstable model output tensor for the next inference process is the model output tensor where inconsistent inference results exist in each inference iteration of the next inference process. If the unstable model output tensor for the next inference process contains the marked current input tensor, then the current input tensor is added to a preset queue. If the unstable model output tensor for the next inference process does not contain the marked current input tensor, then the current input tensor is determined to have consistent results in each inference iteration of the next inference process.

[0064] In one embodiment, the model stability testing method may further include: in response to a release request for the target model, extracting the model weights of the target model and storing them in a storage system; in response to a reproduction request for the target model, retrieving the model weights and model code of the target model from the storage system; converting the model code into an operator computation graph based on the model weights; and reproducing the target model based on the operator computation graph.

[0065] Among them, the target model release request is an instruction initiated by the development or operation personnel to archive and store the trained model.

[0066] A target model reproduction request is a user-initiated instruction to restore a historical version of the model, used to reproduce the model's computational logic and runtime effects.

[0067] Specifically, this application can respond to a release request for a target model, obtain the model weights, model code, and corresponding test data of the target model, and use a model release CI (Continuous Integration) tool to upload the model weights, model code, and test data of the target model to the storage system, identified by the model version of the target model. The storage system can be a distributed storage system, such as HDFS (Hadoop Distributed File System). The model release CI tool is an automated process tool; after the developer submits the model code and weights, the tool automatically triggers processes such as building, testing, and archiving storage.

[0068] In response to a reproduction request for the target model, the model release CI tool will download the corresponding model weights and test data from the storage system, load the model code, and generate the corresponding operator computation graph based on the model weight parsing. The model will then be launched based on the operator computation graph to reproduce the target model. The test data will be used for subsequent model accuracy testing.

[0069] The beneficial effect of this embodiment is that by automatically storing weights and code during release, it avoids the inability to reproduce a previous version of the model due to personnel turnover and data cleanup, thus ensuring the traceability of the model.

[0070] In one embodiment, the model stability testing method may further include: determining the original training model of the target model based on the model weights and model code; using test data to perform inference on the target model and the original training model of the target model respectively to obtain a first result and a second result respectively; calculating the value of a pre-configured accuracy index based on the first result and the second result respectively, and comparing them; and determining the accuracy alignment result between the target model and the original training model of the target model based on the comparison result, so as to perform accuracy testing.

[0071] The original training model refers to the baseline model obtained at the end of the training phase, which has not undergone deployment transformation and retains the complete structure, original weights, and computational logic of the original training framework.

[0072] Accuracy testing refers to the testing process mentioned above, which compares the model with the original training model to a preset accuracy index to verify whether the model's accuracy is aligned.

[0073] Specifically, this application can restore the original training model corresponding to the target model based on the stored model weights and model code, and perform inference on the target model and the original training model using test data respectively to obtain a first result and a second result, wherein the first result is the inference result of the target model and the second result is the result of the original training model.

[0074] Calculate the preset accuracy index value based on the first result and the second result respectively. The preset accuracy index can be mAP (mean Average Precision). Compare the two index values ​​and calculate the difference between them. If the difference is within a preset range, the accuracy of the target model is aligned with that of the original training model; otherwise, they are not aligned.

[0075] The beneficial effect of this embodiment is that by using test data to evaluate the training framework model and the deployment model, it can be confirmed whether the accuracy of the two can be aligned, preventing the inconsistency between the performance of the deployment model and the training framework model, and ensuring the accuracy and reliability of model reproduction.

[0076] In one embodiment, the process of inputting test data into the target model to obtain a first result includes: uploading the model file of the reproduced target model, test data, and model results to the storage system; in response to the model's accuracy test request, submitting a model inference request to the server; the server downloading the model file and test data from the storage system, then performing model inference, uploading the inference result to the storage system; the terminal downloading the inference result uploaded by the server from the storage system, calculating the value of a preset accuracy index based on the inference result, and then comparing it with the value of the preset accuracy index corresponding to the training model to obtain an accuracy alignment result.

[0077] Please refer to Figure 3 for details. In Figure 3, the client first uploads the model file to the distributed file system, then submits the task to the server. The server downloads the model and test data from the distributed file system and performs model inference. After the inference is completed, the result is uploaded back to the distributed file system. Finally, the client downloads the inference result, calculates and compares the value of the preset accuracy index based on the inference result, and realizes the accuracy test of the model.

[0078] In this application, the server may include multiple servers, each with a different GPU installed, and the first result obtained may include multiple results, with each GPU device outputting the corresponding first result.

[0079] Please refer to Figure 4. In Figure 4, the training framework outputs model weights, test data, and the transformed deployment model (ONNX). The model weights and test data are first used to train the deployment accuracy alignment. At the same time, the deployment model is tested on different GPU hardware such as RTX5000, A40, and Orin. The test results of the deployment model and the training model are further compared to perform accuracy testing. The training model is the original training model of the target model. In addition, this application also includes stability testing of the deployment model to finally locate unstable operators.

[0080] Since the deployment model may have different accuracy on different GPU hardware, this application improves the reliability of accuracy testing by verifying accuracy on multiple different GPU devices.

[0081] In one embodiment, the stability testing method may further include: in response to a code merge request in the current code repository, extracting the code modification portion from the code merge request, wherein the current code repository is the repository where the code of the target model is located; when the code modification portion contains information about the model deployment framework corresponding to the target model, performing model stability testing and accuracy testing on all models corresponding to the model deployment framework in the code repository; when the code modification portion contains common code of the target model, performing model stability testing and accuracy testing on each sub-model of the target model; and when the code modification portion is the code of a sub-model of the target model, performing model stability testing and accuracy testing on the sub-model.

[0082] The code repository refers to the version-controlled repository that stores all relevant code for the target model.

[0083] A code merge request is a request made by a developer to the code repository after modifying the code, requesting that the changes be merged into the main branch or target branch.

[0084] A model deployment framework is a tool or platform used to encapsulate trained models into runnable services, responsible for deployment-related logic such as model loading, inference scheduling, and request processing.

[0085] Common code refers to the generic code shared by multiple sub-models in the target model, such as common utility functions, basic network layers, and data preprocessing logic. Modifications to this code may affect all sub-models that depend on it.

[0086] If the target model is a composite model consisting of multiple functional modules, then the sub-model refers to a specific functional module directly corresponding to the part of the code that has been modified.

[0087] Information about the model deployment framework can include the model deployment framework name, version number, deployment configuration parameters, etc.

[0088] Specifically, this application can monitor code merge requests in the target model's code repository, trigger an automated testing process, and extract the code modifications from the merge request, i.e., the code snippets added, modified, or deleted by the developer. If the modifications contain information related to the model's deployment framework, stability and accuracy tests are performed on all models corresponding to that deployment framework in the repository. If the modifications are common code of the target model, stability and accuracy tests are performed on all sub-models of the target model. If the modifications only affect a specific sub-model of the target model, stability and accuracy tests are performed only on that sub-model. The merge is approved based on the test results; if the tests pass, the merge is approved; otherwise, it is rejected with a problem message.

[0089] Please refer to Figure 5. In Figure 5, Model Training Engineer 1 develops model training code (including common parts and Model 1 and Model 2 specific code). Model Training Engineer 2 continues development based on this code. Meanwhile, the Model Deployment Framework Engineer develops the corresponding model deployment framework, which depends on the model training code. If the common parts are modified, stability and accuracy tests need to be performed on Model 1 and Model 2. If only the code for Model 1 is modified, only stability and accuracy tests need to be performed on Model 1. If the information of the model deployment framework is modified, stability and accuracy tests need to be performed on each model corresponding to the model deployment space.

[0090] The advantages of this embodiment are: targeted testing based on the scope of code modifications eliminates the need to test all models, thereby improving testing efficiency and saving resources.

[0091] In a second aspect, this application provides a model stability testing device, as shown in Figure 6. The model stability testing device includes: a current inference module 61, an acquisition module 62, a next inference model 63, a processing module 64, and a determination module 65, wherein: the current inference module 61 is used to execute the current inference process on the target model, the current inference process including multiple inferences with the same test data; the acquisition module 62 is used to acquire the current computing node, the current computing node being the node corresponding to the unstable model output tensor obtained by the current inference process, the unstable model output tensor obtained by the current inference process being the model output tensor with inconsistent inference results in each inference of the current inference process; the next inference module 63 is used to acquire the input tensor of the current computing node and perform the next inference process based on the input tensor of the current computing node, the next inference process including multiple inferences with the same test data; the processing module 64 determines the current computing node as an unstable node when the input tensor of the current computing node is consistent in each inference result of the next inference process; the determination module 65 is used to determine the stability test result of the target model based on the unstable node.

[0092] In one embodiment, the unstable model output tensor obtained in the current inference process includes multiple tensors. The acquisition module 62 can also store the unstable model output tensors obtained in the current inference process into a preset queue, acquire and delete the current tensor from the preset queue, acquire the computing node corresponding to the current tensor, obtain the current computing node, add the variable input to the preset queue when the node type of the current computing node is in the preset whitelist and the number of variable inputs of the current computing node is one, acquire and delete the next tensor from the preset queue, take the next tensor as the current tensor, and return to the step of acquiring the computing node corresponding to the current tensor.

[0093] In one embodiment, the next inference module 63 may, when the node type of the current computing node is not in the preset whitelist and / or the number of variable inputs of the current computing node is not the same, obtain the current input tensor of the current computing node, determine whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed, and skip the current input tensor when the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed, obtain the next input tensor of the current computing node, and return to the step of determining whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been accessed.

[0094] In one embodiment, the next inference module 63 can mark the current input tensor as the model output tensor of the target model when the current input tensor is not a constant, is not the input tensor of the target model, and is not a tensor that has been accessed. This is to adjust the output of the target model, and then use the adjusted model to perform the next inference process based on the current input tensor. The module obtains the unstable model output tensor of the next inference process. The unstable model output tensor of the next inference process is the model output tensor in which inconsistent inference results exist in each inference result of the next inference process. When the unstable model output tensor of the next inference process contains the marked current input tensor, the current input tensor is added to a preset queue. When the unstable model output tensor of the next inference process does not contain the marked current input tensor, it is determined that the current input tensor has consistent results in each inference of the next inference process.

[0095] In one embodiment, the next inference module 63 can also determine whether all input tensors of the current computing node have been traversed. If all input tensors of the current computing node have been traversed and the inference results of the input tensors of the current computing node are consistent in each inference process, the processing module 64 can determine that the current computing node is an unstable node. If there are input tensors of the current computing node that have not been traversed, the next inference module 63 can obtain the next input tensor of the current computing node.

[0096] In one embodiment, the test module 65 may also, in response to the release request of the target model, extract the model weights of the target model and store them in the storage system; in response to the reproduction request of the target model, retrieve the model weights and model code of the target model from the storage system; convert the model code into an operator computation graph based on the model weights; and reproduce the target model based on the operator computation graph.

[0097] In one embodiment, the testing module 65 can also determine the original training model of the target model based on the model weights and model code, use test data to perform inference on the target model and the original training model of the target model respectively, obtain a first result and a second result respectively, calculate the value of the pre-configured accuracy index based on the first result and the second result respectively, compare them, and determine the accuracy alignment result between the target model and the original training model of the target model based on the comparison result, so as to perform accuracy testing.

[0098] In one embodiment, the testing module 65 can also respond to the code merging request of the current code repository, extract the code modification part in the code merging request, where the current code repository is the repository where the code of the target model is located, and when the code modification part contains information about the model deployment framework corresponding to the target model, perform model stability testing and accuracy testing on all models corresponding to the model deployment framework in the code repository, when the code modification part contains common code of the target model, perform model stability testing and accuracy testing on each sub-model of the target model, and when the code modification part is the code of a sub-model of the target model, perform model stability testing and accuracy testing on the sub-model.

[0099] In a third aspect, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the model stability testing method provided in any embodiment of the first aspect of this application.

[0100] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in Figure 7. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a model stability testing method. The display screen of the computer device may be a liquid crystal display (LCD) or an e-ink display. The input device of the computer device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse, etc.

[0101] In a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the model stability testing method provided in any embodiment of the first aspect of this application.

[0102] The computer-readable storage medium may be the computer-readable storage medium in the computer device shown in Figure 7.

[0103] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A model stability testing method, characterized in that, The method includes: executing a current inference process on a target model, the current inference process including multiple inferences with the same test data; obtaining a current computing node, the current computing node being the node corresponding to an unstable model output tensor obtained by the current inference process, the unstable model output tensor being a model output tensor with inconsistent inference results in each inference of the current inference process; obtaining the input tensor of the current computing node, and performing a next inference process based on the input tensor of the current computing node, the next inference process including multiple inferences with the same test data; determining the current computing node as an unstable node when the input tensor of the current computing node is consistent in each inference result of the next inference process; and determining a stable test result of the target model based on the unstable node.

2. The method according to claim 1, characterized in that, The unstable model output tensors obtained from the current inference process include multiple tensors. The method further includes: storing the unstable model output tensors obtained from the current inference process into a preset queue; obtaining the current computing node includes: obtaining and deleting the current tensor from the preset queue; obtaining the computing node corresponding to the current tensor to obtain the current computing node; when the node type of the current computing node is in a preset whitelist and the number of variable inputs of the current computing node is one, adding the variable inputs to the preset queue; obtaining and deleting the next tensor from the preset queue, using the next tensor as the current tensor, and returning to the step of obtaining the computing node corresponding to the current tensor.

3. The method according to claim 2, characterized in that, The step of obtaining the input tensor of the current computing node includes: obtaining the current input tensor of the current computing node when the node type of the current computing node is not in the preset whitelist and / or the number of variable inputs of the current computing node is not the same; the method further includes: determining whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been visited; when the current input tensor is a constant, an input tensor of the target model, or a tensor that has been visited, skipping the current input tensor and obtaining the next input tensor of the current computing node; returning to the step of determining whether the current input tensor is a constant, an input tensor of the target model, or a tensor that has been visited.

4. The method according to claim 3, characterized in that, The step of performing the next inference process based on the input tensor of the current computing node includes: when the current input tensor is not a constant, and the current input tensor is not the input tensor of the target model, and the current input tensor is not a tensor that has been visited, marking the current input tensor as the model output tensor of the target model to adjust the output of the target model; and using the adjusted model to perform the next inference process based on the current input tensor. The method further includes: obtaining the unstable model output tensor of the next inference process, wherein the unstable model output tensor of the next inference process is the model output tensor with inconsistent inference results in each inference result of the next inference process; when the unstable model output tensor of the next inference process contains the marked current input tensor, adding the current input tensor to the preset queue; and when the unstable model output tensor of the next inference process does not contain the marked current input tensor, determining that the current input tensor has consistent results in each inference of the next inference process.

5. The method according to claim 3, characterized in that, Before obtaining the next input tensor of the current computing node, the method further includes: determining whether all input tensors of the current computing node have been traversed; determining the current computing node as an unstable node when the input tensors of the current computing node are consistent in each inference result of the next inference process includes: if all input tensors of the current computing node have been traversed and the input tensors of the current computing node are consistent in each inference result of the next inference process, then the current computing node is determined to be the unstable node; obtaining the next input tensor of the current computing node includes: obtaining the next input tensor of the current computing node when there are input tensors of the current computing node that have not been traversed.

6. The method according to claim 1, characterized in that, The method further includes: in response to a release request for the target model, extracting the model weights of the target model and storing them in a storage system; in response to a reproduction request for the target model, retrieving the model weights and model code of the target model from the storage system; converting the model code into an operator computation graph based on the model weights, and reproducing the target model based on the operator computation graph.

7. The method according to claim 1, characterized in that, The method further includes: determining the original training model of the target model based on the model weights and the model code; performing inference on the target model and the original training model of the target model using test data to obtain a first result and a second result respectively; calculating the value of a pre-configured accuracy index based on the first result and the second result respectively, and comparing them; determining the accuracy alignment result between the target model and the original training model of the target model based on the comparison result, so as to perform accuracy testing.

8. The method according to claim 7, characterized in that, The method further includes: in response to a code merge request in the current code repository, extracting the code modification portion from the code merge request, wherein the current code repository is the repository where the code of the target model is located; when the code modification portion contains information about the model deployment framework corresponding to the target model, performing the model stability test and the accuracy test on all models corresponding to the model deployment framework in the code repository; when the code modification portion contains common code of the target model, performing the model stability test and the accuracy test on each sub-model of the target model; when the code modification portion is the code of the sub-model of the target model, performing the model stability test and the accuracy test on the sub-model.

9. A model stability testing device, characterized in that, The apparatus includes: a current inference module for executing a current inference process on a target model, the current inference process including multiple inferences with the same test data; an acquisition module for acquiring a current computing node, the current computing node being the node corresponding to an unstable model output tensor obtained from the current inference process, the unstable model output tensor being a model output tensor with inconsistent inference results in each inference of the current inference process; a next inference module for acquiring the input tensor of the current computing node and performing a next inference process based on the input tensor of the current computing node, the next inference process including multiple inferences with the same test data; a processing module for determining the current computing node as an unstable node when the input tensor of the current computing node is consistent in each inference result of the next inference process; and a determination module for determining a stable test result of the target model based on the unstable node.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images