Processor test method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a processor testing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining test case data of a target processor, wherein the test case data comprises a test identifier, a test instruction type, a priority, a test instruction description and a generation cue word; processing the test constraint data, the output format data and the test case data through a pre-trained assembly code generation model to generate a test code; standardizing the test code to generate an assembly file; and compiling the assembly file to generate an executable file corresponding to the test instruction, and testing the target processor based on the executable file to obtain a processor test result. By adopting the method, the comprehensive learning of the test instruction can be realized through the assembly code generation model, the full-coverage test code is intelligently generated, and the availability of the generated test code is further ensured.

Description

Technical Field

[0001] This application relates to the field of processor verification technology, and in particular to a processor testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of electronic devices, many types of processors have emerged. Each processor needs to be verified before execution to ensure that its implementation conforms to the open-source instruction set architecture (ISA) specification. Instruction set random testing is the standard for instruction-level verification. Its core is to generate random instruction sequences, run them simultaneously on the processor under test (DUT) and a reference model, and compare the results (such as register values ​​and memory states) for consistency. Related technologies typically use handwritten rules and constraints for verification, which randomly generates multiple redundant instructions, failing to comprehensively cover the processor and resulting in low testing efficiency. Summary of the Invention

[0003] Therefore, it is necessary to provide a processor testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve testing efficiency in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a processor testing method, including:

[0005] Acquire test case data for the target processor, the test case data including test identifier, test instruction type, priority, test instruction description, and generated prompt words;

[0006] The test constraint data, output format data, and test case data are processed using a pre-trained assembly code generation model to generate test code; the test code is then standardized to generate an assembly file.

[0007] The assembly file is compiled to generate an executable file corresponding to the test instructions, and the target processor is tested based on the executable file to obtain the processor test results.

[0008] In one embodiment, the method further includes:

[0009] Based on the target training data, the target base model is fine-tuned to obtain a trained assembly code generation model; the target training dataset includes a test instruction dataset, a processor parameter set, a valid code file dataset, and an invalid code file dataset.

[0010] In one embodiment, the step of performing parameter fine-tuning training on the target base model based on the target training data to obtain a trained assembly code generation model includes:

[0011] The test description data in the test instruction dataset is processed by the target base model to generate predicted instruction data;

[0012] Based on the test instructions in the test instruction dataset, the predicted instruction data, and the preset loss function, the loss data is calculated, and the parameters of the target base model are fine-tuned using the loss data and the training weights corresponding to the test instruction dataset to obtain a trained assembly code generation model.

[0013] In one embodiment, the step of processing test constraint data, output format data, and test case data using a pre-trained assembly code generation model to generate test code includes:

[0014] Based on the test constraint data, output format data, and the test case data, target prompt data is generated.

[0015] The target prompt data is processed to generate test code data; and the test code data is standardized to generate an assembly file.

[0016] In one embodiment, the generated prompt words include positive prompt words and / or negative prompt words; the process of generating target prompt data based on test constraint data, output format data, and the test case data includes:

[0017] Based on the instruction coverage, the test case data, and the test constraint data, instruction constraint data is generated; based on the randomly generated seed identifier, instruction constraint data, and output format data, target prompt data is generated.

[0018] In one embodiment, the target processor is a simulated processor; the step of testing the target processor based on the executable file to obtain processor test results includes:

[0019] The executable file is executed in the simulation processor to obtain simulation test results;

[0020] If the simulation test result is a failure, the test code and the test cases are added to the invalid code file dataset, and if the termination condition is not met, the steps of processing the test constraint data, output format data and the test case data to generate test code are re-executed.

[0021] If the simulated test result is successful, the executable file is used to test the real processor to obtain the real test result; and if the real test result is a failure, the invalid code file dataset is updated based on the failure reason and test code corresponding to the failure result.

[0022] Secondly, this application also provides a processor testing apparatus, comprising:

[0023] The first acquisition module is used to acquire test case data of the target processor. The test case data includes test identifier, test instruction type, priority, test instruction description and generated prompt words.

[0024] The first generation module is used to process test constraint data, output format data, and test case data using a pre-trained assembly code generation model to generate test code; and to standardize the test code to generate an assembly file.

[0025] The second generation module is used to compile the assembly file to generate an executable file corresponding to the test instructions, and to test the target processor based on the executable file to obtain the processor test results.

[0026] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the embodiments of this application.

[0027] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, constitutes the steps in the embodiments of this application.

[0028] Fifthly, this application also provides a computer program product, including a computer program, which, when executed by a processor, represents the steps in the embodiments of this application.

[0029] The aforementioned processor testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product, wherein the method includes: acquiring test case data for the target processor, the test case data including test identifiers, test instruction types, priorities, test instruction descriptions, and generation prompts; processing test constraint data, output format data, and test case data using a pre-trained assembly code generation model to generate test code; standardizing the test code to generate an assembly file; compiling the assembly file to generate an executable file corresponding to the test instructions; and testing the target processor based on the executable file to obtain processor test results. By employing this method, comprehensive learning of test instructions can be achieved through the assembly code generation model, realizing intelligent generation of fully covered test code, further ensuring the usability and practicality of the generated test code. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating a processor testing method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating the model configuration steps in one embodiment;

[0033] Figure 3 This is a flowchart illustrating the model training steps in one embodiment;

[0034] Figure 4 This is a flowchart illustrating the testing steps in one embodiment;

[0035] Figure 5 This is a flowchart illustrating the processor testing method in another embodiment;

[0036] Figure 6 This is a structural block diagram of a processor testing device in one embodiment;

[0037] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0040] In one embodiment, such as Figure 1 As shown, a processor testing method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be a standalone server or a server cluster composed of multiple servers. In this embodiment, the processor testing method includes the following steps:

[0041] Step 102: Obtain test case data for the target processor.

[0042] The test case data includes test identifier, test instruction type, priority, test instruction description, and generated prompts. The target processor can be the type of processor currently requiring instruction testing, such as the processor to be tested determined by the obtained RTL code. The terminal can obtain the RTL code to be tested and determine the processor to be tested after initializing the test framework. Test case data can be descriptive data for testing the processor under test. The test identifier can be a test ID, such as test001. The test instruction type represents the type of instruction to be tested in the test case, such as basic or extension. Priority can include p0, p1, etc., representing the priority of the test case. The test instruction description can be textual description data describing the test instructions to be generated, such as instruction name information. Generation prompts can include positive prompts (prompt_positive_template) and negative prompts (prompt_negative_template). Positive prompts represent the requirements and configurations for the test instructions to be generated, such as the name of the test instruction, the number of test instructions to be generated, and the test coverage of the test instructions. Negative prompts represent content to avoid in the generation of test instructions, i.e., content that is not allowed.

[0043] Optionally, the positive prompt can be "Generate a mixed test sequence of AMOADD.D and AMOAND.D instructions to verify the saturation calculation function"; the negative prompt can be "Uncached memory region: 0x80000000-0xFFFFFFFF".

[0044] Specifically, the terminal can initialize the test framework, pull the RTL code to be tested, and thus obtain the test environment. In essence, the terminal can determine the target processor based on the pulled RTL code, i.e., identify the processor to be tested, and obtain the corresponding test list for that processor. This test list can contain one or more test case data. Optionally, the terminal can process the acquired test case data, for example, it can acquire and process multiple test case data serially; or, if the terminal's capacity is sufficient, the terminal can acquire and process multiple test case data in parallel.

[0045] Step 104: Using a pre-trained assembly code generation model, process the test constraint data, output format data, and test case data to generate test code; and standardize the test code to generate an assembly file.

[0046] The pre-trained assembly code generation model can be obtained by fine-tuning parameters multiple times based on various target training datasets. This assembly code generation model can be a large-scale base model, such as the CodeLlama (7B / 13B) model or the Mistral-7B model, etc. All of these models possess good technical description understanding capabilities and structured code generation capabilities. Test constraint data can be global constraint data, such as the type range of generated instructions or constraint data for test scenarios. Output format data represents the fixed format requirements for the model's output results. Test code can be the model's output result, which can be assembly code. The assembly file can be an assembly file obtained by extracting code blocks from the assembly code and adding necessary headers. The name of this assembly file can be the test identifier in the test case.

[0047] Specifically, the terminal can be configured with a pre-trained assembly code generation model. Test case data can be input into this model, which processes the test case data and test constraint data to generate target hint data. The model then performs code prediction processing on this target hint data to generate the test code corresponding to the test case data, thus obtaining the model output. In this way, the terminal can perform standardized assembly processing on the test code to obtain an assembly file.

[0048] Step 106: Compile the assembly file to generate an executable file corresponding to the test instructions, and test the target processor based on the executable file to obtain the processor test results.

[0049] Among them, assembly files are files that can be compiled, executable files are files that can be executed, the target processor can be the processor to be tested, and the processor test results can be instruction test results obtained by running the executable files corresponding to each test instruction on the processor, which can be test success results, test failure results, etc.

[0050] Specifically, the terminal can compile assembly files, for example, using a pre-defined compilation toolchain to process the compiled files and generate an executable file (Executable and Linking Format, ELF). The terminal can then execute this executable file on the target processor, obtain the execution result, and derive the processor test result based on the execution result. Optionally, the pre-defined compilation toolchain can correspond to the model of the processor under test. For example, if the processor under test is a RISC-V processor, the pre-defined compilation toolchain can be the RISC-V toolchain, specifically riscv64-unknown-elf-gcc.

[0051] In the aforementioned processor testing method, test case data for the target processor is acquired. This data includes test identifiers, test instruction types, priorities, test instruction descriptions, and generated prompts. A pre-trained assembly code generation model processes the test constraint data, output format data, and test case data to generate test code. The test code is then standardized to generate an assembly file. This assembly file is compiled to generate an executable file corresponding to the test instructions. The target processor is then tested based on the executable file to obtain the processor test results. By employing this method, a comprehensive learning of test instructions can be achieved through the assembly code generation model, enabling intelligent generation of fully covered test code and further ensuring the usability and practicality of the generated test code.

[0052] In one embodiment, the processor testing method further includes:

[0053] The target base model is trained by fine-tuning its parameters based on the target training data to obtain a trained assembly code generation model. The target training dataset includes a test instruction dataset, a processor parameter set, a valid code file dataset, and an invalid code file dataset.

[0054] The target training dataset is used for LoRa (parameter fine-tuning) training of the target base model. This dataset can contain multiple datasets for different dimensions of parameter fine-tuning. For example, the test instruction dataset is used to teach the target base model the instruction specification. This dataset can be an instruction set extension LoRa, such as test instruction descriptions and examples extracted from the RISC-V official extension manual (e.g., the "B" extension). These examples can be test code. Besides the "B" extension, this instruction set extension can also include P extensions, V extensions, etc., and different extensions will have corresponding test instruction datasets. The processor parameter set can be a dataset used to define basic information such as the processor's model and attributes. For example, it can be a processor parameter set determined based on the type of processor to be tested, which may include PC address settings, memory accessible address range, memory PMP settings, memory paging mode, memory attribute partitioning, available register names, sizes, and aliases, etc. The valid code file dataset can be scenario-based LoRa, for example, it can include valid assembly code sequences from completed (historical) verification projects, as well as assembly code compiled from (mature) open-source projects that are in use; the invalid code file dataset can include invalid assembly code, unreasonable assembly code sequences, and other content that is not expected to be generated by the assembly code generation model from historical verification projects, and can also include assembly code from failed test results generated by the assembly code generation model trained in this embodiment.

[0055] Specifically, since running LoRA primarily relies on three hardware resources: GPU memory, RAM, and storage, the terminal can pre-configure its hardware environment. GPU VRAM determines the size of the base model and batch size the terminal can load. For example, when choosing the QLoRA (4-bit) strategy, a graphics card with more than 16GB of VRAM should be selected. Correspondingly, since CPU and RAM are responsible for data loading and preprocessing, the RAM size should be sufficient to hold the dataset. Storage stores model weights, datasets, and checkpoints; the terminal can use an SSD. After configuring the terminal's hardware environment, the software environment can be configured. Specifically, the terminal can use FastAPI as the web framework, vLLM as the inference engine, and a multi-LoRA parallel loading mechanism to load various target training datasets: instruction set extended LoRA dataset, environment LoRA dataset, scene LoRA dataset, and reverse LoRA dataset. vLLM is implemented through LoRAManager.

[0056] Optionally, the terminal can assign a unique integer ID (lora_int_id) to each target training dataset and configure a LoRA index table of integer IDs in the terminal's memory. It can also set up level routing, allowing the use of corresponding LoRA weights based on different training requests. These LoRA weights are the weight data used when training the target base model. The requests from each test processor and the corresponding LoRA weights for each target training dataset can be different, determined in memory based on pre-stored correspondences. The LoRA weights for invalid code file datasets can be negative, i.e., less than zero, thus avoiding the generation of data from invalid code file datasets. Optionally, corresponding execution code libraries can also be installed on the terminal, such as Python libraries: torch, transformers, accelerate, peft, bitsandbytes, and trl, etc.

[0057] like Figure 2 The diagram shown can be a flowchart of the API service running the training steps corresponding to the assembly code generation model, such as the access path from the client to the LoRa model:

[0058] The client can call the FastAPI server, which in turn accesses the vLLM engine, and calls the target base model and the LoRa adapter repository. This LoRa adapter repository stores the target training dataset, which may include instruction set extended LoRa datasets, environment LoRa datasets, scene LoRa datasets, and reverse LoRa datasets. Thus, the client can train the target base model by calling each target training dataset from the LoRa adapter repository. After training the target base model on each target training dataset, a trained assembly code generation model can be obtained.

[0059] In this embodiment, the video memory, RAM, and storage configured in the hardware environment can provide the terminal with extremely high throughput and concurrency support for model training.

[0060] In one embodiment, such as Figure 3 As shown, the specific processing steps of "performing parameter fine-tuning training on the target base model based on the target training data to obtain a trained assembly code generation model" include:

[0061] Step 302: Process the test description data in the test instruction dataset using the target base model to generate prediction instruction data.

[0062] The test instruction dataset contains test instruction descriptions and examples, which can be test code data. The target base model may include at least an encoding module and a decoding module.

[0063] Specifically, the terminal can input the test instruction dataset into the target base model, encode the test description data through the encoding module in the target base model to generate a structure vector, and decode the structure vector through the decoding module to generate prediction instruction data, that is, prediction code data.

[0064] Step 304: Based on the test instructions, predicted instructions, and preset loss function in the test instruction dataset, calculate the loss data, and fine-tune the parameters of the target base model using the loss data and the training weights corresponding to the test instruction dataset to obtain the trained assembly code generation model.

[0065] The preset loss function can be the cross-entropy loss function, the MSE loss function, etc. The test instructions in the test instruction dataset are the test code data.

[0066] Specifically, the terminal can calculate the loss data based on the test code and prediction code data in the test instruction dataset. For example, it can process the test code and prediction code data using a preset loss function to obtain the loss data. Then, using the loss data and the training weights corresponding to the test instruction dataset (i.e., the LoRa corresponding to the test instruction dataset collected by the LoRa adapter), the terminal can fine-tune the model parameters of the target base model to obtain the processed target base model. If the preset training completion conditions are not met, the terminal can re-execute the step of processing the test description data in the test instruction dataset using the target base model to generate prediction instruction data based on the processed target base model, until the terminal can determine that the processed target base model is a trained assembly code generation model when the preset training completion conditions are met.

[0067] Optionally, the preset training completion condition could be, for example, that the loss data has converged or remained unchanged, or that the number of training iterations has reached a preset threshold, or that the perplexity of the processed target base model is less than or equal to a preset threshold. The perplexity algorithm can be any algorithm used in the model training process, such as being calculated using exp(eval_loss). This perplexity is used to characterize the uncertainty of the model's predictions. The preset threshold could be 10. For example, a perplexity <10 indicates low uncertainty in the model, meaning the model's performance in the current training iteration is relatively good. Correspondingly, a perplexity between 10 and 50 indicates moderate uncertainty in the model, meaning the model's performance in the current training iteration is good. Conversely, a perplexity >100 indicates high uncertainty in the model, meaning the model's performance in the current training iteration is relatively poor.

[0068] The loss data obtained from the target training dataset is the training loss, which can be automatically calculated during the training of the target base model. It represents the degree of fit of the model to the training data. For example, the loss data should continuously decrease. If the terminal determines that the loss data oscillates too much during the training of the target base model, that is, the difference between the loss data between training batches is large, then the terminal will lower the learning rate during the training of the target base model to obtain an adjusted target base model. Then, the steps described above, which involve processing the test description data in the test instruction dataset through the target base model to generate the prediction instruction data, will be re-executed.

[0069] Optionally, the terminal can also obtain validation sets corresponding to each target training dataset separately. Through a process similar to the training process, an evaluation loss is obtained. This evaluation loss is automatically calculated on the validation set and is the most important indicator representing the model's generalization ability. The evaluation loss should decrease synchronously with the training loss; if it increases, it indicates overfitting. For example, the terminal can input the target training dataset rv_ext_data.jsonl from the above embodiment into the training script to start training, thereby generating a corresponding LOA file, such as riscv_extension_lora. For example, a portion (usually 5-10%) of the target training dataset can be partitioned as a validation set for subsequent internal evaluation, ensuring that the validation set does not overlap with the training set.

[0070] It should be noted that the terminal can train a target base model sequentially using all target training datasets, and after the preset training completion conditions are met during the training process based on each target training dataset, the currently processed target base model can be identified as the pre-trained assembly code generation model. The training and verification processes for each target training dataset are similar to those for the test instruction dataset.

[0071] Optionally, the terminal can sequentially train the target base model based on the test instruction dataset, processor parameter set, valid code file dataset, and invalid code file dataset to obtain a pre-trained assembly code generation model. The training process based on the test instruction dataset is similar to the training process described in the above embodiments. When the terminal determines that the preset training completion conditions are met, it can determine the current target base model as the first base model. Thus, when the target training dataset is the processor parameter set, the first base model includes an encoding module and a decoding module. The process by which the terminal trains the first base model based on the processor parameter set can be as follows: the terminal can input each processor parameter in the processor parameter set into the encoding module to obtain a parameter vector, and decode the parameter vector based on the decoding module to obtain predicted parameters. Then, based on a preset loss function, a loss calculation is performed between the processor parameters and the predicted parameters to obtain a loss value. Subsequently, the model parameters of the first base model are fine-tuned based on the loss value. If the preset training completion conditions are not met, the steps of inputting each processor parameter in the processor parameter set into the encoding module to obtain a parameter vector are repeated until a first base model that meets the preset training completion conditions is obtained. This allows the first base model to learn the processor's parameter sequence, avoiding the generation of test code that does not conform to the processor parameters.

[0072] In this way, the terminal can retrain the first pedestal model, which meets the preset training completion conditions, based on both the valid code file dataset and the invalid code file dataset, either simultaneously or separately. The valid code file dataset is used to enable the first pedestal model to learn correct test code, while the invalid code file dataset is calculated based on negative weights and is used to prevent the first pedestal model from generating code contained in the invalid code file dataset. The terminal can determine that the current first pedestal model is a trained assembly code generation model when both the target training dataset (valid code file dataset) and the target training dataset (invalid code file dataset) meet the preset training completion conditions.

[0073] In this embodiment, by continuously monitoring relevant indicators such as loss data and perplexity during the training process, it is possible to accurately judge whether LoRA training is effective and obtain a professional model with good performance and strong generalization ability while ensuring training efficiency.

[0074] In one embodiment, the specific processing steps of "processing test constraint data, output format data, and test case data using a pre-trained assembly code generation model to generate test code" include:

[0075] Based on the test constraint data, output format data, and test case data, target prompt data is generated; the target prompt data is further processed to generate test code data; and the test code data is standardized to generate assembly files.

[0076] Specifically, for each test case, a unique seed is randomly generated. Based on the instruction coverage, positive and negative prompts in the test case data, and other information, as well as test constraint data, instruction constraint data is generated. This data is then combined with the seed, instruction constraint data, and output format data to obtain the target prompts used to instruct the generation of assembly code—the target prompt data. The terminal can process this target prompt data using the assembly code generation model. The terminal can compile the assembly file, for example, using a preset compilation toolchain to process the compiled file and generate an executable file (Executable and Linking Format, ELF).

[0077] In this embodiment, the intelligence of the instruction generation process can be enhanced by continuously refining the corresponding prompt words, thereby further improving the efficiency of assembly code generation and executable file generation.

[0078] In one embodiment, the generated prompt words include positive prompt words and / or negative prompt words; the specific processing procedure for the step "processing based on test constraint data, output format data, and test case data to generate target prompt data" includes:

[0079] Based on the instruction coverage, test case data, and test constraint data, instruction constraint data is generated; based on the randomly generated seed identifier, instruction constraint data, and output format data, target prompt data is generated.

[0080] The instruction coverage can be the range of generated instructions, such as using only the target instruction set, for example, using only the RV32IMC instruction set. Test constraint data can be requirements for included scenarios, such as "including at least the target number of overflow scenarios." Randomly generated seed identifiers correspond one-to-one with test cases; in subsequent processes, the terminal can use this seed to verify or regenerate the data in the test case, etc. Output format data represents the fixed format requirements for the model output results of the assembly code generation model, which facilitates the parsing of the model output results.

[0081] Specifically, for each test case, different seeds are randomly generated. Based on the instruction coverage, positive and negative prompts in the test case data, and other information, as well as test constraint data, instruction constraint data is generated. Based on the seed, instruction constraint data, and output format data, target prompts, i.e., target prompt data, are obtained.

[0082] Optionally, the output format data can be:

[0083] Return in the following format:

[0084] === TEST BEGIN ===

[0085] .section .text

[0086] .globl _start

[0087] _start:

[0088] [Your assembly code]

[0089] === TEST END ===".

[0090] In this embodiment, the intelligence of the instruction generation process can be enhanced by continuously refining the corresponding prompt words, thereby further improving the efficiency of assembly code generation.

[0091] In one embodiment, the target processor is a simulated processor; the simulated processor can be a standard simulator corresponding to the processor, or it can be a simulated processor identical to the actual processor; such as... Figure 4 As shown, the specific processing steps for the step "testing the target processor based on the executable file and obtaining the processor test results" include:

[0092] Step 402: Execute the executable file in the simulation processor to obtain the simulation test results; if the simulation test results are failures, add the test cases to the invalid code file dataset, and if the termination conditions are not met, re-execute the steps of processing the test constraint data, output format data, and test case data to generate test code.

[0093] The termination condition can be that the number of re-executions reaches a threshold, but this embodiment does not limit the specific value of the threshold. The invalid code file database can be a database for fine-tuning the parameters of the target base model based on negative weights, which can contain test case data and test code of various types of test failures.

[0094] Specifically, the terminal can execute the executable file in the simulated processor to obtain the simulated test results. If the terminal determines that the simulated test result is a failure, it can analyze the test case data corresponding to the executable file to obtain the reason for the failure and upload the failed test case to the invalid code file database. At the same time, if the terminal determines that the test case corresponding to the failure result does not meet the preset termination conditions, it can re-execute the steps of processing the test constraint data, output format data, and test case data to generate test code based on a new seed identifier (i.e., generating a new seed upon re-execution). For example, it can re-execute the step of generating target prompt data to obtain updated target prompt data, and re-generate the updated target prompt data based on the assembly code generation model to generate assembly code, etc., until the test case for which the failure result is currently determined has met the preset termination conditions, such as reaching the preset number of retries. In this way, the terminal can determine that the test corresponding to the test case is terminated.

[0095] Step 404: If the simulation test result is successful, the actual processor is tested through the executable file to obtain the actual test result; and if the actual test result is a failure, the invalid code file dataset is updated based on the failure reason and test code corresponding to the failure result.

[0096] The real processor can be a real RTL environment.

[0097] Specifically, if the terminal determines that the current simulated test result is successful, it can execute the executable file on the real processor to obtain the actual test result. If the actual test result is a failure, the terminal can analyze the test cases corresponding to the failure result to determine the cause of the failure, such as whether the failure is due to a problem with the assembly code generated by the test cases. In this way, the terminal can add the assembly code corresponding to the failure result and the cause of failure to the invalid code file dataset. This allows the invalid code file dataset to be used for reverse training of the assembly code generation model, preventing the model's output from containing any assembly code from the invalid code file dataset.

[0098] Optionally, if the actual test result is determined to be a failed test result, the terminal can also add other constraints corresponding to the failed test result to the invalid code file dataset to achieve further parameter fine-tuning of the code generation model. Other constraints may be, for example, RTL design changes.

[0099] In this embodiment, by executing the executable file using both a simulated processor and a real processor, comprehensive test results of the executable file can be obtained. Test code corresponding to failed test results is added to the invalid code file dataset, improving the effectiveness of model reverse training and further enhancing the validity of the model's output assembly code and verifying test accuracy.

[0100] The following describes in detail the specific implementation steps of the processor testing method described above, using a specific embodiment. RISC-V, as an open-source instruction set architecture (ISA), has experienced rapid ecosystem growth, giving rise to numerous different processor implementations. Ensuring that these processor implementations conform to the ISA specification is crucial; this process is called verification. Instruction set random testing is the gold standard for instruction-level verification. Its core is to generate random instruction sequences, run them simultaneously on the processor under test (DUT) and the gold reference model, and compare the results (such as register values ​​and memory states) for consistency. The processor testing method provided in this embodiment is based on a generative artificial intelligence (AI) model, automatically and intelligently generating high-quality, high-coverage random instruction sequences for verifying the correctness of the RISC-V processor design.

[0101] like Figure 5 As shown, the specific execution process of the processor testing method provided in this embodiment can be as follows:

[0102] The first stage involves data preparation and preprocessing. Specifically, this includes model planning and data sourcing, processing steps, selection of a base model and environment configuration; followed by LoRA training. LoRA training may include selecting a fine-tuning method (LoRA or QLoRA), configuring key hyperparameters, performing training and monitoring, internal evaluation, and calculating perplexity. If the preset training completion conditions are not met, the process returns to the step of selecting the fine-tuning method.

[0103] The datasets used for LoRa training include: a test instruction dataset, i.e., instruction set extension LoRa, which can be test instruction descriptions and examples extracted from the RISC-V official extension manual (such as the "B" extension). It also includes an environment setting LoRa, i.e., a processor parameter set, which is a dataset that limits basic information such as the processor's model and attributes. For example, it can be a processor parameter set determined based on the type of processor to be tested, which may include PC address settings, memory accessible address range, memory PMP settings, memory paging mode, memory attribute partitioning, available register names, sizes, and aliases, etc. It also includes a scenario LoRa, i.e., a valid code file dataset, which may include valid assembly code sequences from completed (historical) verification projects, and assembly code compiled from (mature) open-source projects in use; and a reverse LoRa, i.e., an invalid code file dataset, which may include invalid assembly code, unreasonable assembly code sequences, and other content not expected to be generated by the assembly code generation model from historical verification projects, and may also include assembly code from failed test results generated by the assembly code generation model trained in this embodiment.

[0104] The preprocessing step may involve constructing high-quality (instruction, input, output) triples for each task. Each task can be a different target training dataset, such as data used to generate different types of training data, and adding them to JSON files rv_ext_data.jsonl, env.jsonl, senario.jsonl, and negative.jsonl, respectively, along with the instruction LOA dataset, environment LOA dataset, scene LOA dataset, and reverse LOA dataset in the above embodiments.

[0105] For example, a possible triplet could be as follows: {

[0106] "instruction": "Explain the function and format of the following RISC-V user-defined instructions."

[0107] "input": "Instruction name: P.ADD, Encoding: 0x0600000B",

[0108] "output": "Instruction: P.ADD rd, rs1, rs2\nFunction: Performs a saturation summation of the packed bytes (8 bits) in registers rs1 and rs2, and writes the result to rd.\nEncoding: 0x0600000B, corresponding to the user-defined opcode space.\nNote: This is a SIMD-style packing operation."

[0109] }

[0110] Alternatively, it can also be a triple generated from assembly code (C to ASM):

[0111] {

[0112] "instruction": "Optimize the following C code loop using the custom P.ADD instruction."

[0113] "input": "for (int i=0; i<4; i++) { dst[i] = clamp(src1[i] + src2[i], 0, 255);}",

[0114] "output": " / / Assuming data has been loaded into a1 and a2 in a packaged format\nloop:\nP.ADD a0,a1, a2 # Saturation addition of 4 bytes\n / / ... Storage and loop logic..."

[0115] }

[0116] Alternatively, it can also be a triple for assembly code generation (Natural Language to ASM):

[0117] {

[0118] "instruction": "Write RISC-V assembly code to implement the saturation addition of two 4-pixel values ​​using the new P.ADD instruction."

[0119] "input": "The source pixel values ​​are in registers a1 and a2, and the results are written to a0."

[0120] "output": "P.ADD a0, a1, a2 # a0 = sat_add(a1, a2)"

[0121] }

[0122] Alternatively, it can also be a triple for instruction constraint verification:

[0123] {

[0124] "instruction": "Check if the use of P.ADD in the following instruction sequence is valid."

[0125] "input": "P.ADD x0, a1, a2\ nP.ADD a0, a1, x0",

[0126] "output": "First line: Error. The destination register for P.ADD is x0 (zero register). The result will be discarded, possibly due to a programming error.\nSecond line: Error. The source operand cannot be x0. P.ADD requires actual source data."

[0127] Alternatively, it can be a triple generated from the test cases:

[0128] {

[0129] "instruction": "Generate a boundary test case for the P.ADD instruction to test saturated addition."

[0130] "input": "The input value is close to the maximum value of the byte data type (0xFF).

[0131] "output": "li a1, 0xFF00FF00 # Set a1 to [255, 0, 255, 0]\ nli a2,0x01010101 # Set a2 to [1, 1, 1, 1]\ nP.ADD a0, a1, a2 # Result a0 should be [255, 1,255, 1] ​​(high-order saturation occurs)"

[0132] }

[0133] Phase 2: Deployment model, including configuring the hardware environment (flash, memory, and storage) and the software environment (pytorch / transformers / PEFT) and running API services.

[0134] Specifically, model deployment includes selecting a target base model, such as CodeLlama (7B / 13B) or Mistral-7B. Model deployment also includes selecting the LoRA fine-tuning method, which can be PEFT, updating only a small subset of parameters with near-full-parameter fine-tuning results. Two methods are available: LoRA, which adds a low-rank matrix alongside the original model for fine-tuning, and the matrix can be merged back into the original model after training, resulting in no inference latency; or QLoRA, which performs 4-bit quantization on top of LoRA, making it possible to fine-tune large models on a single consumer-grade GPU (e.g., fine-tuning a 7B model on an RTX 3090). The steps for configuring key hyperparameters include configuring the learning rate, num_train_epochs, etc., as shown in Table 1 below.

[0135] Table 1

[0136]

[0137] In this way, the terminal can train based on the configured target base model and the target training dataset to obtain a trained assembly code generation model.

[0138] Phase Three: Application Model. Specifically, this may include starting the test framework, obtaining the test list, running individual tests, and initiating external validation.

[0139] Optionally, the application model involves connecting to the subsequent chip verification framework after the API service starts, and then performing actual verification testing. Starting the test framework can involve initializing the test framework, pulling the RTL code to be tested, and initializing the test environment. Obtaining the test list can involve retrieving the test cases to be tested, and the appropriate organization method, such as serial or parallel, can be selected based on the capacity of the AI ​​server (terminal). Each test case will include basic information such as ID, type, priority, description, and positive / negative prompts needed to generate the test. Below is an example of test cases test1 and test2:

[0140] {

[0141] "id": "test_001",

[0142] "type": "basic",

[0143] "priority": 1,

[0144] "description": "Testing ADD command",

[0145] "prompt_positive_template": "Generates a test sequence containing 10 ADD instructions, covering all register combinations."

[0146] };

[0147] Or it could also be:

[0148] {

[0149] "id": "test_002",

[0150] "type": "extension",

[0151] "priority": 2,

[0152] "description": "Testing Extended Directive A",

[0153] "prompt_positive_template": "Generates a mixed test sequence of AMOADD.D and AMOAND.D instructions to verify the saturation calculation function."

[0154] "prompt_negative_template": ["Non-cached memory region: 0x80000000-0xFFFFFFFF"]

[0155] }

[0156] This refers to content that is not allowed.

[0157] In this way, the terminal can preprocess the test list, add global constraints and output format limitations, generate the final prompt, and then execute the steps to run individual tests.

[0158] Running a single test could involve sending a structured request to the AI ​​server to retrieve assembly code that meets the verification requirements. The prompts in this structured request include the following data:

[0159] Seed: For example, it can be 'b'; specifically, the submitted prompt must be paired with a randomly generated seed, which will be used to generate the same test sequence for verification after fixing the RTL defect.

[0160] Instruction constraints: For example, ISA extensions and register constraints can be explicitly specified in the prompt. Instruction constraints can be:

[0161] Generate a test sequence with the following requirements:

[0162] - Use only the RV32IMC instruction set

[0163] - Covers all combinations of registers x1-x5

[0164] - Contains at least one overflow scenario

[0165] - Do not use pseudo-instructions.

[0166] The output format can be:

[0167] A fixed format is required for parsing.

[0168] Return in the following format:

[0169] === TEST BEGIN ===

[0170] .section .text

[0171] .globl _start

[0172] _start:

[0173] [Your assembly code]

[0174] === TEST END ===".

[0175] In this way, the terminal can process the assembly code generated by the assembly code generation model, obtain the assembly code generated by the model, and standardize the assembly code into a compilable assembly file. The standardization process may include extracting code blocks and adding necessary headers. The extracted code block is "using regular expression matching === TEST BEGIN ===(.*?) === TEST END ===", and the added necessary headers may be as follows:

[0176] .section .tohost “aw”, @progbits

[0177] .align 6

[0178] .globl tohost

[0179] tohost: .dword 0

[0180] .section .text

[0181] .globl _start

[0182] _start:

[0183] [Assembly code]

[0184] Based on this, the terminal can save the assembly file with a test ID (e.g., test_001.S), and then compile this assembly file to generate an ELF image. Specifically, this can be done using a RISC-V toolchain such as riscv64-unknown-elf-gcc to compile the assembly code into an executable file. The terminal can then test the generated executable file or ELF image using the open-source RISC-V simulator SPIKE. If the test succeeds, proceed to the next step; if the test fails, change the seed and resubmit the prompt, while recording the failed test cases for analysis; if the test still fails after several resubmissions, terminate the test.

[0185] Therefore, the next step could be to run a simulation, which is to verify the behavior in a real RTL environment. For failed test cases, it is necessary to analyze whether the problem lies with the generated test sequence and record the reasons for the failure.

[0186] In addition, the terminal also needs to collect failed test cases or other constraints such as RTL design changes generated in the verification process, add them to the failure dataset for subsequent reverse LoRa training, or adjust the training dataset to train other forward LoRa, and perform daily model retraining iterations to further improve model performance and the efficiency and accuracy of test code generation.

[0187] The processor testing method provided in this embodiment can generate targeted test code, avoiding the generation of meaningless or redundant instructions, improving testing efficiency, and covering deep and complex peripheral cases. It can also improve test coverage convergence speed, avoiding completely random generation methods and reducing the time to achieve high coverage. Furthermore, it achieves flexible high scalability and editability; each new instruction extension (such as P, V, B extensions) only requires adding one instruction test dataset, improving generation and accuracy. It can also construct complex scenarios and automatically generate semantically related instruction sequences (e.g., effectively constructing complex scenarios such as Page Fault, TLB conflict, and precise interrupts).

[0188] In other words, the method provided in this embodiment is a way to automatically and intelligently generate more efficient and targeted random test instructions, which can accelerate the verification cycle of RISC-V processors. Specifically, it can use AI models to learn the semantic, syntactic, and probabilistic relationships between instructions, thereby generating test code that is not only syntactically correct (compliant with ISA specifications) but also semantically rich and more likely to trigger potential errors. Simultaneously, by training different instruction set extensions (LORA, environment setting LOA, and scenario LOA), the accuracy of model generation is improved.

[0189] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0190] Based on the same inventive concept, this application also provides a processor testing apparatus for implementing the processor testing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more processor testing apparatus embodiments provided below can be found in the limitations of the processor testing method described above, and will not be repeated here.

[0191] In one exemplary embodiment, such as Figure 6 As shown, a processor testing apparatus 600 is provided, comprising:

[0192] The first acquisition module 602 is used to acquire test case data of the target processor. The test case data includes test identifier, test instruction type, priority, test instruction description and generated prompt words.

[0193] The first generation module 604 is used to process test constraint data, output format data, and test case data through a pre-trained assembly code generation model to generate test code; and to standardize the test code to generate assembly files.

[0194] The second generation module 606 is used to compile the assembly file, generate the executable file corresponding to the test instructions, and test the target processor based on the executable file to obtain the processor test results.

[0195] Each module in the aforementioned processor testing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0196] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores processor test data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a processor testing method.

[0197] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0198] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0199] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0200] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0201] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0202] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0203] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0204] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A processor testing method, characterized in that, The method includes: Acquire test case data for the target processor, the test case data including test identifier, test instruction type, priority, test instruction description, and generated prompt words; The test constraint data, output format data, and test case data are processed using a pre-trained assembly code generation model to generate test code; and the test code is then standardized to generate an assembly file. The assembly file is compiled to generate an executable file corresponding to the test instructions, and the target processor is tested based on the executable file to obtain the processor test results.

2. The method according to claim 1, characterized in that, The method further includes: The target base model is trained by fine-tuning its parameters based on the target training data to obtain a trained assembly code generation model; the target training dataset includes a test instruction dataset, a processor parameter set, a valid code file dataset, and an invalid code file dataset.

3. The method according to claim 2, characterized in that, The step of fine-tuning the parameters of the target base model based on the target training data to obtain a trained assembly code generation model includes: The test description data in the test instruction dataset is processed by the target base model to generate predicted instruction data; Based on the test instructions in the test instruction dataset, the predicted instruction data, and the preset loss function, the loss data is calculated, and the parameters of the target base model are fine-tuned using the loss data and the training weights corresponding to the test instruction dataset to obtain a trained assembly code generation model.

4. The method according to claim 2, characterized in that, The step of generating test code by processing test constraint data, output format data, and test case data using a pre-trained assembly code generation model includes: Based on the test constraint data, output format data, and the test case data, target prompt data is generated. The target prompt data is processed to generate test code data; and the test code data is standardized to generate an assembly file.

5. The method according to claim 3, characterized in that, The generated prompt words include positive prompt words and / or negative prompt words; The process of generating target prompt data based on test constraint data, output format data, and test case data includes: Based on the instruction coverage, the test case data, and the test constraint data, instruction constraint data is generated. The target prompt data is generated by processing the randomly generated seed identifier, instruction constraint data, and output format data.

6. The method according to claim 1, characterized in that, The target processor is a simulated processor; the test of the target processor based on the executable file to obtain processor test results includes: The executable file is executed in the simulation processor to obtain simulation test results; If the simulation test result is a failure, the test code and the test cases are added to the invalid code file dataset, and if the termination condition is not met, the steps of processing the test constraint data, output format data and the test case data to generate test code are re-executed. If the simulated test result is successful, the executable file is used to test the real processor to obtain the real test result; and if the real test result is a failure, the invalid code file dataset is updated based on the failure reason and test code corresponding to the failure result.

7. A processor testing device, characterized in that, The device includes: The first acquisition module is used to acquire test case data of the target processor. The test case data includes test identifier, test instruction type, priority, test instruction description and generated prompt words. The first generation module is used to process test constraint data, output format data, and test case data using a pre-trained assembly code generation model to generate test code; and to standardize the test code to generate an assembly file. The second generation module is used to compile the assembly file to generate an executable file corresponding to the test instructions, and to test the target processor based on the executable file to obtain the processor test results.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images