Test management method, device, medium, and product

Abstract

The application discloses a test management method, device, medium and product, relates to the technical field of computers, and in the test management method disclosed by the application, two external data acquisition interfaces are added in a parameter manager, the function of supporting explicit parameter storage and automatically capturing all parameters of a current function is supported, and the use process is greatly simplified. After the captured parameters are subjected to type-sensitive formatting, the parameters are stored in a memory cache to realize microsecond-level rapid access, and are simultaneously persisted to a disk through an asynchronous writing mechanism, so that the data safety is ensured. The application is suitable for real parameter capturing and storage of various operators of a model, and provides a data basis for subsequent assetization management and test case generation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a test management method, device, medium, and product. Background Art

[0002] In the context of artificial intelligence, ensuring newly developed operators function properly across a wide range of large models is a fundamental yet thorny challenge engineers often face. Existing technologies suffer from insufficient test parameter integrity, leading to distorted parameter reconstruction during test playback and inability to accurately reproduce test scenarios.

[0003] Therefore, the present application provides a test management method based on real parameter collection to ensure abnormal scenario playback, so as to solve abnormal problems that occur in the actual development environment of the model. Summary of the Invention

[0004] The present application provides a test management method, device, medium and product to at least solve the problems in the related art.

[0005] This application provides a test management method, including:

[0006] In response to detecting that the model starts the inference process, triggering and determining a data capture interface according to the test requirements and the model function encapsulation state to obtain the original test environment data in the model;

[0007] Determining the test environment parameters to be stored contained in the original test environment data according to the data type;

[0008] The test environment parameters to be stored are stored in a designated storage medium and a data calling interface is provided to call the test environment parameters to be stored to generate test cases.

[0009] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned test management methods when executing the computer program.

[0010] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned test management methods are implemented.

[0011] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned test management methods when executed by a processor.

[0012] In the test management method disclosed in this application, two external data acquisition interfaces are added to the parameter manager to support the explicit specification of parameter storage and the automatic capture of all parameters of the current function, greatly simplifying the usage process. After the captured parameters are formatted with type sensitivity, they are stored in the memory cache to achieve microsecond-level fast access, and are persisted to the disk through an asynchronous write mechanism to ensure data security. It is suitable for capturing and storing real parameters of various operators of the model, providing a data foundation for subsequent asset management and test case generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] Figure 1 A flow chart of a test management method provided in an embodiment of the present application;

[0015] Figure 2 A schematic diagram of a test management method provided in an embodiment of the present application;

[0016] Figure 3 This is a test management system architecture diagram provided in an embodiment of the present application;

[0017] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0020] As mentioned in the background technology disclosure, engineers often face a fundamental yet thorny challenge in the early stages of developing large AI models: ensuring that newly developed operators function correctly across a wide range of large models. For example, when developing an attention mechanism operator, programmers must verify its output across dozens of input parameters, varying sequence lengths, multiple precision formats, and various memory layouts. The combination of just two dimensions, sequence length and precision format, can generate tens of thousands of test cases.

[0021] In existing technologies, developers typically employ primary testing methods that suffer from parameter combination explosion and insufficient test coverage. One approach relies on manually written static test cases, pre-written checklists for every possible scenario; this approach struggles to cover the complex parameter combinations encountered during actual runtime. Another approach utilizes randomly generated parameters for fuzz testing, but this approach can repeatedly test irrelevant parameters and miss critical edge cases. Another approach, based on log replay testing, only records function signatures and basic call counts, failing to fully preserve key metadata such as tensor shapes, memory layouts, and data types. Debugging tools primarily focus on memory error detection and lack automated test case generation capabilities. Traditional unit testing frameworks rely on manually constructed test data, making it difficult to cover the real-world runtime scenarios of complex operators. While symbolic execution can explore program paths, its computational overhead when processing large tensor loads limits its practicality. Even more problematic, when an operator encounters a problem during deployment, developers often struggle to reproduce the full set of parameters that actually occurred, making debugging time-consuming and labor-intensive.

[0022] Therefore, there is an urgent need for a test management method that can achieve real-world parameter collection and playback of abnormal scenarios to resolve abnormal problems that arise in the actual model development environment. It is understandable that this application adopts a plug-in architecture design. By accessing the parameter manager module, it can quickly adapt to the parameter characteristics of various AI accelerators (such as NPUs and optical computing chips). It can also cover the expansion needs of diverse test scenarios while maintaining the stability of the core architecture, achieving the technical effect of "core stability and peripheral flexibility."

[0023] The embodiment of the present application provides a test management method implemented in an artificial intelligence development and testing project, which realizes data formatting and storage of real parameters during model test operation, such as Figure 1 As shown, the method disclosed in the embodiment of the present application is applied to a test management system, which automatically captures and stores key parameters of deep learning operators in the model during runtime to further generate test cases for test environment playback, including:

[0024] S1, in response to detecting that the model starts the inference process, triggering the determination of the data capture interface according to the test requirements and the model function encapsulation state to obtain the original test environment data in the model.

[0025] In a specific implementation scenario, the step S1 specifically comprises:

[0026] When the model starts the inference process, that is, there is a deep learning operator running in the current model, the data capture process is started to obtain the key parameters in the model. It can be understood that in the embodiments of the present application, monitoring of multiple models can be simultaneously implemented, and the number of models to be monitored and the specific selection of the models are pre-set and bound by the tester.

[0027] Specifically, the original test environment data includes model-related data, that is, before triggering the determination of the data capture interface according to the test requirements and the model function encapsulation state and calling the corresponding data capture interface to obtain the original test environment data in the model, the initialization function is called to record the model-related data in response to detecting that the model starts the inference process; wherein the model-related data includes the model type, the model version and the test time. The model-related data can also include labels and input metadata for describing the characteristics of the model. The present application does not limit the specific data types and quantities contained in the metadata, which are pre-defined by the tester before the start of the test management and bound in the parameter manager. Specifically, by embedding a specified function in the parameter manager, the initialization of the parameter manager is realized after the inference process is started to obtain the model-related data, such as the “_init_func_record” initialization function. The present application initializes the parameter manager and obtains the model-related data of the corresponding model, which provides a basis for subsequent construction of the assetized management database, so as to improve the manageability of the database.

[0028] Specifically, the data capture interface includes a first capture interface and a second capture interface; wherein the first capture interface is an interface that needs to be manually defined to find the amount, and the second capture interface is an automatic data capture interface. The determination of the data capture interface according to the test requirements and the model function encapsulation state to obtain the original test environment data in the model comprises:

[0029] Determine the function encapsulation state of the model, which includes the encapsulation state and the non-encapsulation state; in response to detecting that the function encapsulation state of the model is the encapsulation state, determine that the data capture interface is the first capture interface. In response to detecting that the function encapsulation state of the model is the non-encapsulation state, detect the test requirements, which include full parameter requirements and non-full parameter requirements; in response to detecting that the test requirements are full parameter requirements, determine that the data capture interface is the second capture interface; in response to detecting that the test requirements are non-full parameter requirements, determine that the data capture interface is the first capture interface. It can be understood that when the function encapsulation state of the model is the encapsulation state, it is impossible to directly find the function definition, which makes it difficult to directly locate the operator position. At this time, it is necessary to enable the first capture interface and manually enter the variable parameters that need to be saved; further, in order to reduce data storage pressure and avoid the acquisition of unnecessary data; the embodiment of the present application also considers starting the corresponding capture interface according to the actual amount of parameters to be obtained, that is, to implement cautious calling of the second capture window to reduce the waste of computing resources, thereby further improving data capture efficiency.

[0030] After determining that the data capture interface is the first capture interface, the method for acquiring the original test environment data specifically includes:

[0031] The first capture interface is called to obtain the original test environment data in the model that matches the pre-stored test environment parameters; the first capture interface is embedded in the parameter manager of the model; the pre-stored test environment parameters are parameters pre-transmitted to the first capture interface to indicate the first capture interface to capture.

[0032] Specifically, take the rotary_embedding operator (a position encoding algorithm) as an example to illustrate:

[0033] ParamRecorder.save(positions=positions, query=query, key=key,

[0034] head_size=self.head_size,

[0035] cos_sin_cache=self.cos_sin_cache,

[0036] is_neox_style=self.is_neox_style,

[0037] func_name="rotary_embedding")

[0038] ops.rotary_embedding(positions, query, key, self.head_size, self.cos_sin_cache, self.is_neox_style).

[0039] Because the aforementioned "rotary_embedding" is already encapsulated within "ops.", its operator definition is difficult to locate. Therefore, the first capture interface (ParamRecorder.sav) is triggered for data capture. However, the raw test environment data to be obtained is determined by pre-entered, stored test environment parameters, such as "positions=positions." Although data capture through the first capture interface requires manual entry of saved variable parameters, overall intrusion and performance loss are minimized. After the normal model test is run, the parameter manager embedded in the first capture interface captures the raw test environment data.

[0040] After determining that the data capture interface is the second capture interface, the above-mentioned method for acquiring the original test environment data specifically includes: calling the second capture interface to acquire the original test environment data within the model; wherein, the embedding location of the second capture interface is determined by the operator definition location in the model's parameter manager. In other words, this application provides a minimally invasive usage method. When performing data capture in the parameter manager, it is only necessary to locate the algorithm definition location and add the implementation code of the second capture interface. Then, the model test is run normally to complete the data capture. The parameter management controller will further generate the test environment parameters to be stored based on the captured original test environment data, so as to further store it in the designated storage medium to generate a storage file.

[0041] Taking the "flash_attn_varlen_func" operator as an example, "flash_attn_varlen_func" is a complex operator currently used in various models. It is complex to implement and has about 25 input parameters. During the development or debugging process, if only randomly generated test data or manually self-defined parameters are used, it is difficult to fully cover the parameter examples in the actual usage scenario, and there may be hidden dangers to correctness. The second capture interface disclosed in the embodiment of the present application is used to automatically capture the original test environment parameters matched by "flash_attn_varlen_func" when the model is running. The implementation code is as follows:

[0042] def flash_attn_varlen_func(

[0043] q,

[0044] k,

[0045] v,

[0046] max_seqlen_q,

[0047] cu_seqlens_q,

[0048] max_seqlen_k,

[0049] cu_seqlens_k=None, # only used for non-paged prefill

[0050] seqused_k=None,

[0051] q_v=None,

[0052] dropout_p=0.0,

[0053] softmax_scale=None,

[0054] causal=False,

[0055] window_size: Optional[List[int]] = None,

[0056] softcap=0.0, # 0.0 means deactivated

[0057] alibi_slopes=None,

[0058] deterministic=False,

[0059] return_attn_probs=False,

[0060] block_table=None,

[0061] return_softmax_lse=False,

[0062] out=None,

[0063] # FA3 Only

[0064] scheduler_metadata=None,

[0065] q_descale=None,

[0066] k_descale=None,

[0067] v_descale=None,

[0068] # Version selector

[0069] fa_version: int = DEFAULT_FA_VERSION, ):

[0071] ParamRecorder.save_all()

[0072] S2. Determine the test environment parameters to be stored contained in the original test environment data according to the data type.

[0073] After obtaining the original test environment data using the first capture interface or the second capture interface, the data needs to be further processed to generate the test environment parameters to be stored. The above method determines the test environment parameters to be stored contained in the original test environment data based on the data type, and completes the processing of key parameters in the original test environment data through type-sensitive formatting functions. It supports lossless capture of mixed-type parameters such as tensors and scalars, and retains the complete computing context, including:

[0074] In response to detecting the presence of a scalar in the original test environment data, determine the data type of the scalar parameter and the matching scalar parameter value as the test environment parameter to be stored; detect whether there is a tensor in the original test environment data according to the tensor parameter dictionary; in response to detecting the presence of a tensor in the original test environment data, extract and determine the metadata corresponding to the tensor as the test environment parameter to be stored.

[0075] The tensor parameter dictionary is pre-entered into the parameter manager. The parameter manager performs a type check on each parameter value contained in the original test environment data based on the tensor parameter dictionary to identify the tensors contained therein. Then, for the identified tensors, its core descriptive metadata is extracted, including but not limited to shape, stride, data type (Dtype), and device location (Device). For shape, the dimensional structure of the tensor is obtained and converted into a standard list format for serialization; for stride, the stride information of the tensor in memory that reflects the physical layout of the data is read; for data type, the numerical precision of the tensor (such as float32, float16, int64) is obtained and standardized into a string representation; for recording the computing device where the tensor is located (such as cpu, cuda:0), its physical location is clarified.

[0076] S3. Store the test environment parameters to be stored in a designated storage medium and provide a data calling interface to call the test environment parameters to be stored to generate test cases.

[0077] In the method disclosed in the embodiment of the present application, a memory-disk dual cache mechanism is adopted, and the above-mentioned designated storage medium includes memory and hard disk; microsecond-level fast writing is achieved through memory cache (records), and asynchronous persistence is performed to disk (such as dump_to_disk_optimized) to ensure data security, wherein the above-mentioned test environment parameters to be stored in the disk are used to generate a test environment parameter file, wherein the test environment parameter file is stored in the form of a yaml file, that is, the test environment parameters to be stored are stored in a yaml file in a preset storage format to complete persistent storage, providing a basis for realizing offline analysis operators, and providing an implementation basis for subsequent generation of test cases to reproduce specific test scenarios, thereby more efficiently locating and solving problems. Among them, yaml (YAML Ain't Markup Language) is a highly readable data sequence that focuses on data rather than tags. It uses a practical and concise syntax to represent complex data structures and has high readability. It is widely used in configuration file writing and data serialization.

[0078] It is understandable that when the method disclosed in the embodiment of the present application monitors the reasoning process of the model, a model reasoning process may have multiple threads, such as the case of multiple cards running, etc. Different threads may go to the data storage at the same time, and in order to ensure that only one thread can store at the same time, the orderliness of data storage is ensured. The present application proposes to set a global thread lock to ensure multi-threaded security, and ultimately perform double-cache mechanism storage under the premise of zero blocking of the main thread, so as to take into account the dual needs of real-time debugging (fast memory access) and data persistence (disk storage). At the same time, data source storage is an asynchronous operation, and the continued operation of the main program will not be affected while waiting, so the performance loss can basically be ignored.

[0079] The above-mentioned storage of the test environment parameters to be stored in the designated storage medium is specifically implemented as follows: in response to detecting that the global lock is available, asynchronously storing the test environment parameters to be stored in the memory and disk to generate a test environment parameter file; in response to detecting that the global lock is unavailable, inserting the storage thread of the target environment parameters matched by the model into the waiting queue. One of the storage threads corresponds to an inference thread in the above-mentioned inference process. It can be understood that one inference process corresponds to one YAML file. After the storage threads corresponding to each inference thread are completed, the test environment parameters to be stored corresponding to each storage thread are collected to form a test environment parameter file.

[0080] Among them, after the storage thread of the target environment parameters matched by the model is inserted into the waiting queue, it also includes: real-time monitoring of the global lock; in response to monitoring that the global lock is available, determining the target storage thread according to the queue order of the waiting queue; running the target storage thread to realize asynchronous storage of the test environment parameters to be stored contained in the target storage thread to the memory and disk to generate a test environment parameter file.

[0081] The acquisition and storage of the test environment parameters to be stored is implemented using a parameter manager. The parameter manager is a tool for recording and managing model parameter changes or historical values, and is commonly used in scenarios such as parameter status tracking and debugging optimization. In this embodiment of the present application, the parameter manager is customized and modified to add two external data call interfaces (i.e., a first capture interface and a second capture interface). These interfaces support explicit parameter storage and automatic capture of all parameters for the current function, significantly simplifying its use. The captured parameters are formatted with type sensitivity and stored in a memory cache for microsecond-level fast access. They are also persisted to disk via an asynchronous write mechanism to ensure data security. This design balances high performance with ease of use and is suitable for capturing and storing real parameters for various operators in the model. It provides a data foundation for subsequent asset management and test case generation. This enables the acquisition, persistence, and formatting of real data during deep learning operator testing, further enabling the automatic generation of test cases covering all parameter combinations in actual operation. This reduces test parameter reconstruction error to 0%, addressing the existing issue of parameter distortion in log playback. It also significantly improves test case generation efficiency and enables accurate reproduction of abnormal scenarios in production environments, shortening troubleshooting time.

[0082] In a specific implementation scenario, the pseudo code of the parameter manager that implements the above functions is as follows:

[0083] class ParamRecorder: # Class-level shared state;

[0084] _records = {} # Memory cache;

[0085] {func_name: [param_entry1, param_entry2]};

[0086] _lock = threading.Lock() # thread lock;

[0087] _file_index = {} # File iteration counter: {func_name: current_iter};

[0088] # === Main Interface ===

[0089] @classmethod

[0090] def save(cls, func_name: str, **kwargs): """First capture interface named parameters"""

[0091] with cls._lock: # thread synchronization

[0092] cls._init_func_record(func_name) # Initialize the structure and save the model metadata

[0093] entry = cls._format_params(kwargs) # Type-sensitive formatting parameter data

[0094] cls._records[func_name].append(entry) # memory cache

[0095] cls._dump_to_disk_optimized(func_name, entry) # Asynchronous write to disk

[0096] @classmethod

[0097] def save_all(cls,): """Second capture interface named parameter"""

[0098] frame = inspect.currentframe().f_back # parameter capture

[0099] func_name = frame.f_code.co_name

[0100] with cls._lock:

[0101] cls._init_func_record(func_name) # Initialize the structure and save the model metadata

[0102] params = cls._capture_parameters(frame) # Intelligent parameter binding

[0103] entry = cls._format_parameters(params) # Type-sensitive formatting parameter data

[0104] cls._records[func_name].append(entry) # memory cache

[0105] cls._dump_to_disk_optimized (func_name, entry) # Asynchronous write to disk

[0106] # === Core Functions ===

[0107] @classmethod

[0108] def _capture_parameters(cls, frame) ->Dict:

[0109] """Dynamically bind parameters"""

[0110] @classmethod

[0111] def _init_func_record(cls, frame) ->Dict:

[0112] """Structure initialization, test metadata extraction"""

[0113] @classmethod

[0114] def _format_parameters(cls, raw_params: Dict) ->Dict:

[0115] """Type-sensitive formatting, including processing of tensors, scalars, and corresponding lists"""

[0116] @classmethod

[0117] def _dump_to_disk_optimized (cls, func_name: str, entry: Dict):

[0118] """Asynchronous disk write"""

[0119] It can be understood that the parameter manager is embedded with both the first capture interface and the second capture interface, and the two interfaces are called according to actual conditions.

[0120] To illustrate with a specific embodiment, when the parameter manager calls the first capture interface to acquire and store data, you can see that params_rotary_embedding.yaml has completely saved the test metadata and the actual parameter information of each call to rotary_embedding; the specific implementation code is as follows:

[0121] _meta:

[0122] model: "baichuan2-13B-chat"

[0123] git_commit: "a1b2c3d" # Not actual, for example

[0124] capture_time: "2025-06-20T14:32:18Z" # Not actual, for example

[0125] params:

[0126] - block_size: 32

[0127] - max_num_seqs: 256

[0128] - temperature: 0.8 # Not actual, just an example

[0129] tags: ["cuda_mode"] # Tags can be added manually

[0130] Iteration_1:

[0131] positions:

[0132] type: tensor

[0133] shape:

[0134] - 265

[0135] stride:

[0136] - 1

[0137] dtype: torch.int64

[0138] query:

[0139] type: tensor

[0140] shape:

[0141] - 265

[0142] - 16

[0143] - 64

[0144] stride:

[0145] - 1024

[0146] - 64

[0147] - 1

[0148] dtype: torch.bfloat16

[0149] key:

[0150] type: tensor

[0151] shape:

[0152] - 265

[0153] - 1

[0154] - 64

[0155] stride:

[0156] - 576

[0157] - 64

[0158] - 1

[0159] dtype: torch.bfloat16

[0160] head_size: 64

[0161] cos_sin_cache:

[0162] type: tensor

[0163] shape:

[0164] - 163840

[0165] - 64

[0166] stride:

[0167] - 64

[0168] - 1

[0169] dtype: torch.bfloat16

[0170] is_neox_style: false

[0171] ---

[0172] Iteration_2:

[0173] …

[0174] ---

[0175] Iteration_3:

[0176] For example, for the X operator, the params_X.yaml file is generated when the parameter manager calls the second capture interface for data acquisition and storage, and the specific implementation code is as follows:

[0177] _meta:

[0178] model: "baichuan2-13B-chat"

[0179] git_commit: "a1b2c3d" # Non-real, example

[0180] capture_time: "2025-06-20T14:32:18Z"# Non-real, example

[0181] params:

[0182] - block_size: 32

[0183] - max_num_seqs: 256

[0184] - temperature: 0.8 # Non-real, example

[0185] tags: ["varlen"] # Manually added tags

[0186] Iteration_1:

[0187] q:

[0188] type: tensor

[0189] shape:

[0190] - 265

[0191] - 16

[0192] - 192

[0193] stride:

[0194] - 3072

[0195] - 192

[0196] - 1

[0197] dtype: torch.bfloat16

[0198] k:

[0199] type: tensor

[0200] shape:

[0201] - 265

[0202] - 16

[0203] - 192

[0204] stride:

[0205] - 3072

[0206] - 192

[0207] - 1

[0208] dtype: torch.bfloat16

[0209] v:

[0210] type: tensor

[0211] shape:

[0212] - 265

[0213] - 16

[0214] - 192

[0215] stride:

[0216] - 3072

[0217] - 192

[0218] - 1

[0219] dtype: torch.bfloat16

[0220] max_seqlen_q: 265

[0221] cu_seqlens_q:

[0222] type: tensor

[0223] shape:

[0224] - 2

[0225] stride:

[0226] - 1

[0227] dtype: torch.int32

[0228] max_seqlen_k: 265

[0229] cu_seqlens_k:

[0230] type: tensor

[0231] shape:

[0232] - 2

[0233] stride:

[0234] - 1

[0235] dtype: torch.int32

[0236] seqused_k: null

[0237] q_v: null

[0238] dropout_p: 0.0

[0239] softmax_scale: 0.1147213867929261

[0240] causal: true

[0241] window_size: null

[0242] softcap: 0.0

[0243] alibi_slopes: null

[0244] deterministic: false

[0245] return_attn_probs: false

[0246] block_table: null

[0247] return_softmax_lse: false

[0248] out: null

[0249] scheduler_metadata: null

[0250] q_descale: null

[0251] k_descale: null

[0252] v_descale: null

[0253] fa_version: 2

[0254] ---

[0255] Iteration_2:

[0256] …

[0257] ---

[0258] Iteration_3:

[0259] …

[0260] In some implementation scenarios, in order to further improve the efficiency of data capture, the present application also proposes to set a deduplication function in the parameter manager and provide a requirement input interface, wherein the tester can input the test requirements into the requirement input interface. The test requirements include the first requirement and the second requirement, wherein the first requirement is used to indicate that various parameters and the number of times the parameters occur need to be recorded during the model operation, and the second requirement is used to indicate that various parameters and the number of times the parameters occur do not need to be recorded during the model operation. The parameter manager runs the deduplication function in response to the test requirement being the second requirement. It is worth noting that the deduplication function does not run by default, that is, no cleaning is required. By setting the deduplication logic in the data capture stage, the data is cleaned after hitting special requirements, and the duplicate parameters are removed, which greatly reduces the subsequent storage pressure and improves the data storage efficiency. The deduplication function can be implemented by constructing a filtering function, and the specific logic of the function is:

[0261] "is_duplicate = _check_paramters(params)

[0262] if is_duplicate:

[0263] continue

[0264] else:

[0265] cls._dump_to_disk_optimized (func_name, entry) # Asynchronous writing

[0266] else:

[0267] ”.

[0268] After the above step S3 is performed to store the test environment parameters to be stored to the specified storage medium, the method further comprises constructing an asset management library:

[0269] In response to a first preset condition trigger, a data calling interface is called to obtain the test environment parameter file saved in the specified storage medium; the test environment parameter files are merged according to a merging condition to generate a test management file; and the test management file is stored to the asset management library with the model-related data contained in the test management file as an index.

[0270] It can be understood that the method disclosed in the present application generates a plurality of test environment parameter files corresponding to a plurality of inference processes in a model, and of course generates a plurality of test environment parameter files for different models. In order to improve the comprehensiveness of the constructed asset management library and reduce redundant operations, the present application controls the data in the asset management library to be updated regularly or periodically by setting a first preset condition. That is, the above-mentioned preset condition can be set to reach a certain time period, for example, 12h, or directly set to all inference processes in the specified model being completed, and the specific first preset condition is not limited by the present application and can be set by a person skilled in the art according to actual needs. The merging condition is set by the developer according to actual needs, such as merging different operators of a unified model, the same operator of different models, all contents under the same test version, and the like, and the specific merging condition is not limited by the present application.

[0271] The asset management library further provides a retrieval interface, and the asset management library parses a retrieval condition received by the retrieval interface to obtain valid retrieval values matched with the model-related data in response to the retrieval condition; and the asset management library finds and outputs a test management file matched with the valid retrieval values in response to the valid retrieval values obtained by parsing. It can be understood that the finally output file will retain the original parameter structure. That is, in the asset management library disclosed in the present application, the non-existent parameter attribute can be skipped as needed to avoid interruption of retrieval. In addition, the retrieval interface also supports multi-level parameter retrieval, such as supporting nested attribute access (such as q.dtype) and top-level attribute (such as max_seqlen_q), and supporting comparison operator retrieval (such as >, <, ==,!=) to realize flexible comparison.

[0272] After the above-mentioned use of the yaml format to complete the persistence and formatted storage of runtime data, the present application constructs a lightweight database (i.e., an asset management library), establishes an index, realizes fast retrieval of use cases across versions and across scenarios, and realizes efficient reuse of test assets. The pseudo code is as follows:

[0273] class ParamRecorder:

[0274] # Here only the function content corresponding to this module is represented

[0275] # === Main Interface ===

[0276] @classmethod

[0277] def merge_yaml_files(file_pattern: str, output_file: str) ->None:

[0278] """Merge all YAML files that match the specified pattern"""

[0279] merged_data = {}

[0280] file_count = 0

[0281] for filepath in glob.glob(file_pattern)

[0282] # Find all matching files

[0283] #Merge data to avoid key name conflicts

[0284] # Warning information, etc.

[0285] # Write new yaml

[0286] @staticmethod

[0287] def filter_cases(func_name: str, filter_conditions: Dict[str, Any],output_file: str) ->None:

[0288] """Filter test cases in YAML files based on conditions"""

[0289] # Build file matching pattern

[0290] file_pattern = f"*{func_name}*.yaml"

[0291] matched_cases = {}

[0292] # Parsing filter conditions

[0293] conditions = []

[0294] for key, value in filter_conditions.items():

[0295] if '.' in key: # handle nested attributes such as q.dtype

[0296] parts = key.split('.')

[0297] conditions.append((parts, value))

[0298] else: # Process top-level attributes such as max_seqlen_q

[0299] conditions.append(([key], value))

[0300] # Loop through all matching files

[0301] # Write new yaml.

[0302] In the specific implementation scenario of finding all attention test cases using bfloat16 and sequence length > 256, the specific implementation code is as follows:

[0303] ParamRecorder.filter_cases(

[0304] func_name="flash_attn_varlen_func",

[0305] filter_conditions={

[0306] "max_seqlen_q": ">256",

[0307] "q.dtype": "torch.bfloat16"

[0308] },

[0309] output_file="long_sequences_bf16.yaml" )

[0311] In the specific implementation scenario of finding abnormal use cases of a specific CUDA version, the specific implementation code is as follows:

[0312] ParamRecorder.filter_cases(

[0313] func_name="flash_attn",

[0314] filter_conditions={

[0315] "_meta.cuda_version": "==11.7",

[0316] "tags": "edge_case" # Metadata tag filtering

[0317] },

[0318] output_file="cuda117_edge_cases.yaml"

[0319] ).

[0320] It is understood that the implementation method for creating and managing the above-mentioned asset management library can be encapsulated in a separate module or not, and this application does not limit this. The encapsulated code module can be embedded in the parameter manager and displayed as an asset management library interface to be called in response to the first preset condition.

[0321] By building a searchable and traceable asset management library, we can use pre-defined search APIs to filter and query parameter data by model version (such as Llama and Baichuan), semantics (such as "max_seqlen_q > 32"), nested attributes (such as "q.dtype = bfloat16"), and metadata (such as CUDA version and test tags). This creates a shareable and reusable test asset library. This library also allows for accurate reproduction of abnormal scenarios in production environments, shortening troubleshooting time.

[0322] Furthermore, after executing the above step S3 to store the test environment parameters to be stored in the designated storage medium, the present application also proposes automatic generation of test cases, specifically including:

[0323] In response to the triggering of the second preset condition, the above-mentioned data call interface is called to obtain the target test environment parameter file saved in the specified storage medium, or the retrieval interface of the asset management library is called to obtain the target test environment parameter file; data preprocessing is performed on the target test environment parameter file to obtain the target test environment parameter set; the target test environment parameter set is imported into the specified position in the test template according to the parameter type to generate the target test case. It can be understood that the generation of the above-mentioned test case is triggered in response to the second preset condition, and the second preset condition triggers the generation of the test case when the test case generation instruction is detected. The test case generation instruction also includes input parameters determined by the open tester, such as model name, model version, and specific parameters such as seqlen_q>1024. The method disclosed in the embodiment of the present application obtains a matching target test environment parameter file from the data call interface or obtains a matching target test environment parameter file from the retrieval interface according to the input parameters contained in the test case generation instruction.

[0324] The above-mentioned data preprocessing of the target test environment parameter file to obtain the target test environment parameter set includes:

[0325] Extract the initial test environment parameter set from the target test environment parameter file, the initial test environment parameter set includes iterative parameters in a nested structure; that is, implement iterative parameter extraction, such as processing the Iteration_N nested structure to extract the original parameters therein. Calculate the hash values ​​of the multiple parameters contained in the initial test environment parameter set; based on the hash value, eliminate the duplicate parameters contained in the initial test environment parameter set to generate the target test environment parameter set. Deduplication based on hash values ​​is a conventional technical means in this field, and this application will not be expanded here. Further, yaml.dump(sorted_keys=True) can be implemented to ensure that the order of parameters does not affect the hash value to improve the accuracy of deduplication. The retention of unique adoption number configurations is achieved through the above-mentioned hash value deduplication mechanism, such as using yaml.dump(sorted_keys=True)+MD5 to generate a unique fingerprint of the parameters for deduplication.

[0326] This application automatically extracts typical parameter combinations using a deep deduplication algorithm. Combined with templated code generation technology, it converts real parameters into executable test code, enabling automated test case production. Test cases come with multiple pre-set templates and support automatic adaptation to mainstream testing frameworks like pytest and unittest, enabling a "record-as-test" automated pipeline.

[0327] It is understood that the above-mentioned method for implementing the automatic generation of test cases can be encapsulated in a separate module or not, and this application does not limit this. The encapsulated code module can be embedded in the parameter manager to display the test case generation interface so as to be called in response to the second preset condition.

[0328] Among them, the pseudo code automatically generated by the test case is as follows:

[0329] class ParamRecorder:

[0330] # === Test Generation Interface ===

[0331] @classmethod

[0332] def generate_test(cls, func_name: str, config: Dict[str, Any] =None):

[0333] """

[0334] Generating unit tests from parameter logging files

[0335] parameter:

[0336] func_name: target function name (corresponding to params_{func_name}.yaml)

[0337] config: Generate configuration {

[0338] "output_dir": "tests", # output directory

[0339] "template": "pytest", # Template type

[0340] "enable_assert": True, # whether to generate assertions

[0341] "import_modules": ["torch"] # Modules to import

[0342] }

[0343] """

[0344] # 1. Parameter file loading and verification

[0345] # 2. Parameter deduplication (based on deep hashing)

[0346] unique_params = []

[0347] seen_hashes = set()

[0348] for params in all_params:

[0349] param_hash = hash(yaml.dump(params, sort_keys=True))

[0350] if param_hash not in seen_hashes:

[0351] seen_hashes.add(param_hash)

[0352] unique_params.append(params)

[0353] # 3. Test code template selection

[0354] # 4. Parameter initialization code generation

[0355] # 5. Test main frame generation

[0356] # 6. File output and verification

[0357] In addition, the method disclosed in the embodiment of the present application supports associating formula description files to enhance the readability of the generated test cases. That is, when the test cases are automatically generated, a pre-generated formula.yaml file is automatically read, which records the corresponding formula content in a formatted format. This helps testers understand how to call operators when generating test code. The content in formula.yaml is as follows:

[0358] test: out = a * b + x

[0359] rotary_embedding: out = ops.rotary_embedding(positions, query, key,head_size, cos_sin_cache, is_neox_style)

[0360] flash_attn_varlen_func: out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(

[0361] q, k, v,

[0362] out,

[0363] cu_seqlens_q,

[0364] dummy_cu_seqlens_k if cu_seqlens_k is None else cu_seqlens_k,

[0365] seqused_k,

[0366] None,

[0367] block_table,

[0368] alibi_slopes,

[0369] max_seqlen_q,

[0370] max_seqlen_k,

[0371] dropout_p,

[0372] softmax_scale,

[0373] False,

[0374] causal,

[0375] real_window_size[0],

[0376] real_window_size[1],

[0377] softcap,

[0378] return_softmax_lse and dropout_p>0,

[0379] None, )

[0381] The target test environment parameter set is imported into the specified location in the test template according to the parameter type to generate the target test case. In the specific implementation scenario, this includes: automatically adding the required module imports (such as torch), intelligent tensor construction, and test loop framework according to the parameter type. For example, after obtaining params_flash_attn_varlen_func.yaml in the first step, simply call ParamRecorder.generate_test("flash_attn_varlen_func") to generate a test case (test_flash_attn_varlen_func.py:). The specific implementation code is as follows:

[0382] mport torch

[0383] # Add more modules here.

[0384] # 本单元测试的是 flash_attn_varlen_func

[0385] # 参数初始化

[0386] unique_params = [

[0387] {

[0388] 'q': torch.randn([265, 16, 192], dtype=torch.bfloat16).as_strided([265, 16, 192], [3072, 192, 1]),

[0389] 'k': torch.randn([265, 16, 192], dtype=torch.bfloat16).as_strided([265, 16, 192], [3072, 192, 1]),

[0390] 'v': torch.randn([265, 16, 192], dtype=torch.bfloat16).as_strided([265, 16, 192], [3072, 192, 1]),

[0391] 'max_seqlen_q': 265,

[0392] 'cu_seqlens_q': torch.randn([2], dtype=torch.int32).as_strided([2],[1]),

[0393] 'max_seqlen_k': 265,

[0394] 'cu_seqlens_k': torch.randn([2], dtype=torch.int32).as_strided([2],[1]),

[0395] 'seqused_k': None,

[0396] 'q_v': None,

[0397] 'dropout_p': 0.0,

[0398] 'softmax_scale': 0.1147213867929261,

[0399] 'causal': True,

[0400] 'window_size': None,

[0401] 'softcap': 0.0,

[0402] 'alibi_slopes': None,

[0403] 'deterministic': False,

[0404] 'return_attn_probs': False,

[0405] 'block_table': None,

[0406] 'return_softmax_lse': False,

[0407] 'out': None,

[0408] 'scheduler_metadata': None,

[0409] 'q_descale': None,

[0410] 'k_descale': None,

[0411] 'v_descale': None,

[0412] 'fa_version': 2,

[0413] },

[0414] {

[0415] 'q': torch.randn([31739, 16, 192], dtype=torch.bfloat16).as_strided([31739, 16, 192], [3072, 192, 1]),

[0416] 'k': torch.randn([31739, 16, 192], dtype=torch.bfloat16).as_strided([31739, 16, 192], [3072, 192, 1]),

[0417] 'v': torch.randn([31739, 16, 192], dtype=torch.bfloat16).as_strided([31739, 16, 192], [3072, 192, 1]),

[0418] 'max_seqlen_q': 280,

[0419] 'cu_seqlens_q': torch.randn(

[120] , dtype=torch.int32).as_strided(

[120] , [1]),

[0420] 'max_seqlen_k': 280,

[0421] 'cu_seqlens_k': torch.randn(

[120] , dtype=torch.int32).as_strided(

[120] , [1]),

[0422] 'seqused_k': None,

[0423] 'q_v': None,

[0424] 'dropout_p': 0.0,

[0425] 'softmax_scale': 0.1147213867929261,

[0426] 'causal': True,

[0427] 'window_size': None,

[0428] 'softcap': 0.0,

[0429] 'alibi_slopes': None,

[0430] 'deterministic': False,

[0431] 'return_attn_probs': False,

[0432] 'block_table': None,

[0433] 'return_softmax_lse': False,

[0434] 'out': None,

[0435] 'scheduler_metadata': None,

[0436] 'q_descale': None,

[0437] 'k_descale': None,

[0438] 'v_descale': None,

[0439] 'fa_version': 2,

[0440] }, ]

[0442] for params in unique_params:

[0443] # 解包参数

[0444] q=params['q'], k=params['k'], v=params['v'], max_seqlen_q=params['max_seqlen_q'], cu_seqlens_q=params['cu_seqlens_q'], max_seqlen_k=params['max_seqlen_k'], cu_seqlens_k=params['cu_seqlens_k'], seqused_k=params['seqused_k'], q_v=params['q_v'], dropout_p=params['dropout_p'], softmax_scale=params['softmax_scale'], causal=params['causal'], window_size=params['window_size'], …

[0445] # 此处添加测试公式

[0446] # The formula is: out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd( q, k,v, out, cu_seqlens_q, dummy_cu_seqlens_k if cu_seqlens_k is None else cu_seqlens_k, seqused_k, None, block_table, alibi_slopes, max_seqlen_q, max_seqlen_k, dropout_p, softmax_scale, False, causal, real_window_size[0], real_window_size[1], softcap, return_softmax_lse and dropout_p>0, None, )

[0447] # Run the test

[0448] output = flash_attn_varlen_func(q, ...)

[0449] # Automatically generated assertions (optional)

[0450] assert output.shape == expected_shape

[0451] The test case code generated based on the above content complies with PEP8 specifications and can be directly integrated into the CI / CD process. This ensures 100% coverage of test case parameters and scenarios in the actual application environment, significantly improving the efficiency of creating regression tests from real-world operational data. This significantly reduces the cost of building and maintaining test cases, providing a library of high-quality, accurately reproducible test cases. It is also scalable and supports custom configuration of various test templates.

[0452] like Figure 2 As shown, in a specific embodiment, the present application breaks through the limitations of traditional large-model back-end operator development and testing in terms of parameter integrity, use case coverage, and parameter asset management. First, the important data and test metadata of the target operator parameters are captured with low intrusion and low performance loss at runtime and are persisted and formatted for storage. Then, two usage modules, asset management library and test case generation, were developed to achieve 100% true reproduction of test scenarios, an order of magnitude improvement in test efficiency, and full life cycle management of test assets. Through the organic combination of lossless capture of runtime parameters and automated test asset production lines, a high-coverage test case library can be automatically constructed based on actual operation data.

[0453] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0454] Based on the test management method embodiment disclosed above, this application also provides a test management system, such as Figure 3 As shown, it includes a data capture and storage module 310, an asset management library module 320 and a test case generation module 330;

[0455] Among them, the data capture and storage module 310 is used to respond to the detection of the model to start the inference process, trigger the determination of the data capture interface according to the test requirements and the model function encapsulation status to obtain the original test environment data in the model; determine the test environment parameters to be stored contained in the original test environment data according to the data type; store the test environment parameters to be stored in the designated storage medium and provide a data calling interface to call the test environment parameters to be stored to generate test cases.

[0456] Among them, the asset management library module 320 is used to respond to the first preset condition trigger, call the data call interface to obtain the test environment parameter file saved in the specified storage medium; merge the test environment parameter file to generate a test management file according to the merging condition; and store the test management file in the asset management library using the model-related data contained in the test management file as an index.

[0457] Among them, the test case generation module 330 is used to respond to the second preset condition trigger, call the data call interface to obtain the target test environment parameter file saved in the specified storage medium, or call the retrieval interface of the asset management library to obtain the target test environment parameter file; perform data preprocessing on the target test environment parameter file to obtain the target test environment parameter set; import the target test environment parameter set to the specified location in the test template according to the parameter type to generate the target test case.

[0458] For the description of the features in the above-mentioned corresponding test management system embodiment, reference can be made to the relevant description of the corresponding embodiment of the test management method, and no further details will be given here.

[0459] An embodiment of the present application further provides an electronic device, comprising: one or more processors; and a memory associated with the one or more processors, the memory being configured to store program instructions. When the program instructions are read and executed by the one or more processors, the program instructions perform the following operations:

[0460] in, Figure 4The electronic device architecture is shown as an example, and may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a bus 430.

[0461] Among them, the processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0462] The memory 420 can be implemented in the form of ROM (Read Only Memory, writable memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the execution of the electronic device 400, and a basic input and output system (BIOS) 422 for controlling the low-level operations of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and an icon font processing system 425, etc. can also be stored. The above-mentioned icon font processing system 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0463] The input / output interface 413 is used to connect to input / output modules to enable information input and output. The input / output modules can be configured as components within the device (not shown) or externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, and various sensors, while output devices may include a display, speaker, vibrator, indicator light, and the like.

[0464] The network interface 414 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.).

[0465] The bus 430 comprises a pathway for transmitting information between the various components of the device, such as the processor 410 , the video display adapter 411 , the disk drive 412 , the input / output interface 413 , the network interface 414 , and the memory 420 .

[0466] In addition, the electronic device 400 can also obtain information on specific collection conditions from the virtual resource object collection condition information database for use in condition judgment.

[0467] It should be noted that although the above device only shows the processor 410, video display adapter 411, disk drive 412, input / output interface 413, network interface 414, memory 420, bus 430, etc., in the specific implementation process, the device may also include other components necessary for normal execution. In addition, those skilled in the art will understand that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0468] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above-mentioned test management method embodiments when running.

[0469] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0470] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above test management method embodiments are implemented.

[0471] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned test management method embodiments are implemented.

[0472] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0473] The above is a detailed introduction to a test management method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A test management method, characterized in that: The method comprises: In response to detecting that the model has started an inference process, triggering and determining a data capture interface based on test requirements and model function encapsulation status to obtain original test environment data within the model, the data capture interface including a first capture interface and a second capture interface, the first capture interface being an interface requiring manual definition of search volume, and the second capture interface being an automated data capture interface; Determining the test environment parameters to be stored contained in the original test environment data according to the data type; Storing the test environment parameters to be stored in a designated storage medium and providing a data calling interface to call the test environment parameters to be stored to generate a test case; wherein, triggering and determining the data capture interface to obtain the original test environment data according to the test requirements and the model function encapsulation status includes: Determine the function encapsulation state of the model, where the function encapsulation state of the model includes an encapsulation state and a non-encapsulation state; In response to detecting that the function encapsulation state of the model is an encapsulation state, determining that the data capture interface is a first capture interface; In response to detecting that the function encapsulation state of the model is a non-encapsulation state, detecting the test requirement, the test requirement including a full parameter requirement and a non-full parameter requirement; In response to detecting that the test requirement is a full parameter requirement, determining that the data capture interface is a second capture interface; In response to detecting that the test requirement is a non-full parameter requirement, the data capture interface is determined to be a first capture interface.

2. The test management method according to claim 1, characterized in that: The method for obtaining the original test environment data includes: Calling a first capture interface to obtain original test environment data in the model that matches pre-stored test environment parameters; wherein the first capture interface is embedded in the parameter manager of the model; The pre-stored test environment parameters are parameters that are pre-transmitted to the first capture interface to instruct the first capture interface to capture.

3. The test management method according to claim 2, characterized in that: The method for obtaining the original test environment data further includes: Calling the second capture interface to obtain the original test environment data in the model; The embedding position of the second capture interface is determined by the operator definition position in the parameter manager of the model.

4. The method according to claim 1, wherein The step of determining the test environment parameters to be stored contained in the original test environment data according to the data type includes: In response to detecting that a scalar parameter exists in the original test environment data, determining a data type of the scalar parameter and a matching scalar parameter value as a test environment parameter to be stored; Detecting whether a tensor exists in the original test environment data according to a tensor parameter dictionary; In response to detecting that the tensor exists in the original test environment data, metadata corresponding to the tensor is extracted and determined as a test environment parameter to be stored.

5. The method according to claim 1, wherein The storing the test environment parameters to be stored in a designated storage medium includes: In response to detecting that the global lock is available, asynchronously storing the test environment parameters to be stored in the memory and the disk to generate a test environment parameter file; In response to detecting that the global lock is unavailable, a storage thread of the target environment parameter matched by the model is inserted into a waiting queue.

6. The method according to claim 5, characterized in that After inserting the storage thread of the target environment parameter matched by the model into a waiting queue, the method further includes: Real-time monitoring of global locks; In response to monitoring that the global lock is available, determining a target storage thread according to a queue order of the waiting queue; The target storage thread is run to asynchronously store the test environment parameters to be stored contained in the target storage thread to the memory and the disk to generate a test environment parameter file.

7. The method according to claim 1, characterized in that The original test environment data also includes model-related data. Before triggering the determination of the data capture interface based on the test requirements and the model function encapsulation status to obtain the original test environment data in the model, the method further includes: In response to detecting that the model starts an inference process, calling an initialization function to record data related to the model; The model-related data includes model type, model version and test time.

8. The method according to claim 7, characterized in that After storing the test environment parameters to be stored in the designated storage medium, the method further includes constructing an asset management library: In response to a first preset condition trigger, calling the data calling interface to obtain the test environment parameter file stored in the designated storage medium; Merge the test environment parameter files to generate a test management file according to the merging conditions; The test management file is stored in the asset management library using the model-related data contained in the test management file as an index.

9. The method according to claim 8, characterized in that The asset management library provides a retrieval interface, and the method further includes: The asset management library, in response to the search condition received by the search interface, parses the search condition to obtain a valid search value that matches the model-related data; In response to the valid search value obtained through parsing, the asset management library searches for and outputs a test management file that matches the valid search value.

10. The test management method according to claim 9, characterized in that: The method further comprises: In response to a second preset condition trigger, calling the data calling interface to obtain the target test environment parameter file stored in the designated storage medium, or calling the retrieval interface of the asset management library to obtain the target test environment parameter file; Performing data preprocessing on the target test environment parameter file to obtain a target test environment parameter set; Import the target test environment parameter set into a specified location in the test template according to the parameter type to generate a target test case.

11. The method according to claim 10, characterized in that The performing data preprocessing on the target test environment parameter file to obtain a target test environment parameter set includes: Extracting an initial test environment parameter set from the target test environment parameter file, wherein the initial test environment parameter set includes iteration parameters in a nested structure; Calculating hash values ​​of multiple parameters included in the initial test environment parameter set; Based on the hash value, duplicate parameters included in the initial test environment parameter set are eliminated to generate a target test environment parameter set.

12. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the test management method according to any one of claims 1 to 11 when executing the computer program.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the test management method according to any one of claims 1 to 11 when executed by a processor.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the test management method according to any one of claims 1 to 11 are implemented.

Images