Logical device interface specification protocol testing method, device, equipment and storage medium

Through artificial intelligence content generation technology, fine-tuning training of the solid-state drive protocol is carried out to automatically generate test cases, solving the problems of high testing costs and low coverage of the solid-state drive protocol, achieving efficient and comprehensive testing coverage, and improving product quality.

CN118467398BActive Publication Date: 2024-11-08INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410918956.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-08
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

In the prior art, the SSD protocol testing cost is high and the test coverage is low, and there is a risk of missing test scenarios. Especially for small businesses and research institutions, purchasing and maintaining professional test fixtures increases the cost, and testing engineers understand the protocol standards. Deviation leads to insufficient coverage of test cases.

Method used

Through artificial intelligence content generation technology, the non-volatile memory host controller interface specification protocol is fine-tuned and trained based on a general inference model to generate and optimize test cases, and use the protocol to understand the model to automatically generate test cases to improve test coverage and efficiency.

Benefits of technology

It reduces testing costs, improves test coverage and efficiency, ensures the integrity, effectiveness and reliability of test cases, reduces the risk of missing test scenarios, and improves the quality of solid-state drive products.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118467398B_ABST
    Figure CN118467398B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device, electronic device and storage medium for testing a logic device interface specification protocol, which relates to the field of computer technology. The method comprises obtaining a non-volatile memory host controller interface specification protocol to be tested, inputting a protocol understanding model, and generating a test case; testing the non-volatile memory host controller interface specification protocol to be tested based on the test case, and outputting a protocol test result; wherein the protocol understanding model is obtained based on fine-tuning and training a general reasoning model, and evaluating and optimizing the test cases generated in each round. The present invention obtains a protocol understanding model in the field of non-volatile memory host controller interface specification testing by fine-tuning the general model, generates and optimizes test cases based on the protocol understanding model, improves test efficiency and coverage, and thus improves the quality of non-volatile memory host controller interface specification solid-state hard disk products.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for testing a logic device interface specification protocol. Background Art

[0002] Since the Non-Volatile Memory express (NVMe) protocol for accessing enterprise-level solid-state drives (SSDs) supports the connection of many different types of storage devices to computer systems, protocol-related tests are crucial. Through protocol testing, the interoperability between NVMe SSDs from different manufacturers and various host systems, operating systems, and storage controllers can be ensured; through protocol testing, it can be checked whether the SSD complies with the specification requirements of the NVMe specification to ensure that it is compatible with other devices that follow the same standard; when users purchase NVMe SSDs, they usually refer to the results of protocol testing to evaluate the performance and reliability of the product, thereby enhancing their confidence in the product. In the actual use of products, some problems may occur due to incomplete understanding of the protocol, resulting in the omission of test scenarios. How to avoid such problems is a great challenge for test engineers. Therefore, how to realize the automatic generation of standard test cases for the NVMe protocol and improve the coverage of test scenarios has become a problem that needs to be solved urgently. Currently, the following two solutions are mainly used to verify the non-volatile memory host controller interface specification solid-state drive protocol test: Solution 1: Use test fixtures to test the non-volatile memory host controller interface specification solid-state drive protocol; Solution 2: Test engineers study the non-volatile memory host controller interface specification protocol standard documents, design test cases, and use tools to write automated test scripts to verify the protocol. For Solution 1, since professional test fixtures are usually expensive, for some small enterprises or research institutions, purchasing and maintaining these test fixtures will increase cost expenses. In addition, these test fixtures have complex operation interfaces and setting processes, and require professional technicians to operate and maintain them. At the same time, the test fixtures cannot fully cover the requirements in the protocol standard, and the test coverage is low. For Solution 2, test engineers have subjective initiative and deviations in their understanding of the protocol standard, resulting in insufficient test case coverage and the omission of certain test scenarios. Summary of the invention

[0003] The present invention provides a logic device interface specification protocol testing method, device, electronic device and storage medium, which are used to solve the defects of traditional solid-state hard disk protocol testing, such as high cost, low test coverage and risk of missing test scenarios.

[0004] The present invention provides a method for testing a logic device interface specification protocol, comprising:

[0005] Obtaining a non-volatile memory host controller interface specification protocol to be tested, inputting the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model, and generating a test case;

[0006] Testing the non-volatile memory host controller interface specification protocol to be tested based on the test case, and outputting a protocol test result;

[0007] The protocol understanding model is obtained based on fine-tuning training of a general reasoning model and evaluation and optimization based on test cases generated in each round.

[0008] According to the logical device interface specification protocol testing method provided by the present invention, the protocol understanding model is based on fine-tuning training of a general reasoning model, including:

[0009] Constructing a non-volatile memory host controller interface specification protocol data set, wherein the non-volatile memory host controller interface specification protocol data set includes key information of the non-volatile memory host controller interface specification protocol and key information of a device interaction command line tool;

[0010] Fine-tune the general inference model based on key information of the non-volatile memory host controller interface specification protocol and key information of the device interaction command line tool to generate an incremental tuning parameter file;

[0011] The incremental tuning parameter file is loaded into the general reasoning model, and some parameters in the general reasoning model are fine-tuned.

[0012] According to the logic device interface specification protocol testing method provided by the present invention, the construction of a non-volatile memory host controller interface specification protocol data set includes:

[0013] Acquire a non-volatile memory host controller interface specification, and acquire a completion queue entry and a submission queue entry in the non-volatile memory host controller interface specification, wherein the completion queue entry is acquired based on a command execution result, and the submission queue entry is acquired based on a command submission result;

[0014] Building a protocol knowledge graph based on the completion queue entries and the submission queue entries;

[0015] Supplement the key commands and command parameters of the device interaction command line tool to the protocol knowledge graph;

[0016] A non-volatile memory host controller interface specification protocol dataset is constructed based on the supplemented protocol knowledge graph.

[0017] According to the logical device interface specification protocol testing method provided by the present invention, the protocol understanding model is based on fine-tuning training of a general reasoning model, and further includes:

[0018] Based on the protocol standard document, typical protocol test cases in each feature module are obtained, and the running process of the non-volatile memory host controller interface specification protocol is described in the format of (passage, question, answer), where passage is used to describe a feature or command of the protocol, question is used to describe a specific question, and answer is used to describe a detailed answer to the question;

[0019] Input (passage, question, answer) into a universal reasoning model so that the universal reasoning model can learn the operation process of the non-volatile memory host controller interface specification protocol and fine-tune some parameters in the universal reasoning model.

[0020] According to the logic device interface specification protocol testing method provided by the present invention, the non-volatile memory host controller interface specification protocol to be tested is input into a protocol understanding model to generate a test case, including:

[0021] Traverse the protocol standard documents of the interface specification protocol of the non-volatile memory host controller to be tested, and extract partial texts one by one;

[0022] Identify the type of each partial text, perform structural processing on the partial text, and identify the commands related to the protocol in the partial text;

[0023] Fill the commands related to the protocol in the partial text into the corresponding type of Prompt template;

[0024] The filled Prompt template is input into the protocol understanding model to generate the corresponding test case.

[0025] According to the logical device interface specification protocol testing method provided by the present invention, the filled Prompt template is input into the protocol understanding model to generate a corresponding test case, including:

[0026] If the type of the partial text is a paragraph, the Prompt template guides the protocol comprehension model to generate a lightweight data exchange format test case including detailed test steps and expected results;

[0027] If the type of the partial text is a form, the Prompt template will guide the protocol understanding model to generate a test case for a specific command parameter bit setting;

[0028] If the type of the partial text is a picture, the Prompt template will guide the protocol comprehension model to generate a test case based on the picture content.

[0029] According to the logic device interface specification protocol testing method provided by the present invention, each round of generated test cases is evaluated, including:

[0030] The test case is evaluated for completeness, specifically including outputting the completeness of the test case according to whether the current test case covers the test cases corresponding to the paragraphs, forms and pictures respectively.

[0031] According to the logic device interface specification protocol testing method provided by the present invention, each round of generated test cases is evaluated, including:

[0032] The effectiveness of the test case is evaluated, specifically including outputting the effectiveness of the test case according to whether the command parameter execution result of the device interactive command line tool is correct.

[0033] According to the logic device interface specification protocol testing method provided by the present invention, each round of generated test cases is evaluated, including:

[0034] The reliability of the test case is evaluated, specifically including executing the test case under different test environments and conditions, and verifying the reliability of the output test case according to the test case execution results.

[0035] According to the logic device interface specification protocol testing method provided by the present invention, each round of generated test cases is optimized, including:

[0036] Screen the test cases generated in each round according to their completeness, effectiveness, and reliability to obtain high-quality test cases;

[0037] Optimizing the non-volatile memory host controller interface specification protocol data set with the high-quality test cases;

[0038] Using the optimized non-volatile memory host controller interface specification protocol data set to train the protocol understanding model, to obtain an optimized protocol understanding model;

[0039] Regenerate test cases according to the optimized protocol understanding model, use the regenerated test cases to test the non-volatile memory host controller interface specification protocol to be tested, and output protocol test results.

[0040] According to the logic device interface specification protocol testing method provided by the present invention, the test case includes a case title, pre-conditions, test steps and expected results, and optimizing each round of generated test cases further includes:

[0041] If the test case title, preconditions, test steps and expected results of the newly generated test case are the same as those of the existing test case, the newly generated test case is determined to be a duplicate test case, and the duplicate test case is deleted from the optimized non-volatile memory host controller interface specification protocol data set.

[0042] The present invention also provides a logic device interface specification protocol testing device, comprising:

[0043] An input module, used for obtaining the interface specification protocol of the non-volatile memory host controller to be tested, inputting the interface specification protocol of the non-volatile memory host controller to be tested into a protocol understanding model, and generating a test case;

[0044] A test module, used for testing the non-volatile memory host controller interface specification protocol to be tested based on the test case, and outputting the protocol test result;

[0045] The protocol understanding model is obtained based on fine-tuning training of a general reasoning model and evaluation and optimization based on test cases generated in each round.

[0046] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the logic device interface specification protocol testing method as described in any one of the above items when executing the program.

[0047] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-mentioned logic device interface specification protocol testing methods.

[0048] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned logic device interface specification protocol testing methods.

[0049] The present invention provides a method, device, electronic device and storage medium for testing a logic device interface specification protocol. The method obtains a non-volatile memory host controller interface specification protocol to be tested, inputs the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model, and generates a test case. The non-volatile memory host controller interface specification protocol to be tested is tested based on the test case, and a protocol test result is output. The protocol understanding model is obtained by fine-tuning and training a general reasoning model, and evaluating and optimizing the test cases generated in each round. The artificial intelligence content generation technology is used to perform test optimization on a non-volatile memory host controller interface specification solid-state hard disk product. Starting from the non-volatile memory host controller interface specification protocol standard, a protocol understanding model in the field of non-volatile memory host controller interface specification testing is obtained by fine-tuning the general model. Test cases are generated and optimized based on the protocol understanding model to improve test efficiency and coverage, thereby improving the quality of non-volatile memory host controller interface specification solid-state hard disk products. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is one of the flow charts of the logical device interface specification protocol testing method provided by the embodiment of the present invention;

[0052] Figure 2 This is the second flow chart of the logic device interface specification protocol testing method provided by the embodiment of the present invention;

[0053] Figure 3 This is the third flow chart of the logic device interface specification protocol testing method provided by the embodiment of the present invention;

[0054] Figure 4 This is a fourth flow chart of the logic device interface specification protocol testing method provided by an embodiment of the present invention;

[0055] Figure 5 It is a functional structure diagram of a logic device interface specification protocol test device provided by an embodiment of the present invention;

[0056] Figure 6 It is a functional structure diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Figure 1 A flowchart of a method for testing a logical device interface specification protocol provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the logical device interface specification protocol testing method provided by the embodiment of the present invention includes:

[0059] Step 101, obtaining a non-volatile memory host controller interface specification protocol to be tested, inputting the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model, and generating a test case;

[0060] Step 102: Test the non-volatile memory host controller interface specification protocol to be tested based on the test case, and output the protocol test result; wherein the protocol understanding model is based on fine-tuning and training the general reasoning model, and is obtained by evaluating and optimizing the test cases generated in each round.

[0061] Traditional verification of non-volatile memory host controller interface specification solid-state drive protocol testing mainly adopts the following two solutions: Solution 1: Use test fixtures to test the non-volatile memory host controller interface specification solid-state drive protocol; Solution 2: Test engineers study the standard documents of the non-volatile memory host controller interface specification protocol, design test cases, and use tools to write automated test scripts to verify the protocol. For Solution 1, since professional test fixtures are usually expensive, for some small enterprises or research institutions, purchasing and maintaining these test fixtures will increase cost expenses. In addition, these test fixtures have complex operation interfaces and setting processes, and require professional technicians to operate and maintain them. At the same time, the test fixtures cannot fully cover the requirements in the protocol standard, and the test coverage is low. For Solution 2, test engineers have subjective initiative and deviations in their understanding of the protocol standard, resulting in insufficient test case coverage and the omission of certain test scenarios.

[0062] With the development of artificial intelligence generated content (AIGC) technology, computers can automatically generate various forms of digital content. In the field of testing, the outbreak of AIGC has triggered extensive research and discussion on its application possibilities in software testing. Traditional software testing methods often require a lot of manpower and time investment, and the introduction of AIGC technology may bring about very big changes in the field of testing. The advantages of AIGC in testing lie in its efficient automation capabilities, rapid learning capabilities, and the ability to process large-scale data. However, the current application model is a general model, lacking in-depth understanding and specialization in specific fields. In specific business scenarios such as non-volatile memory host controller interface specification solid-state drive products, general models perform poorly in processing professional terms, specific rules or business logic, and the test cases generated by the general model are not accurate.

[0063] The logic device interface specification protocol testing method provided by the embodiment of the present invention obtains the non-volatile memory host controller interface specification protocol to be tested, inputs the non-volatile memory host controller interface specification protocol to be tested into the protocol understanding model, generates test cases; tests the non-volatile memory host controller interface specification protocol to be tested based on the test cases, and outputs the protocol test results; wherein the protocol understanding model is obtained based on fine-tuning and training the general reasoning model, and evaluating and optimizing the test cases generated in each round. The content generation technology of artificial intelligence is used to optimize the test of non-volatile memory host controller interface specification solid-state hard disk products. Starting from the non-volatile memory host controller interface specification protocol standard, the protocol understanding model in the field of non-volatile memory host controller interface specification testing is obtained by fine-tuning the general model, and the test cases are generated and optimized based on the protocol understanding model to improve the test efficiency and coverage, thereby improving the quality of non-volatile memory host controller interface specification solid-state hard disk products.

[0064] Based on any of the above embodiments, the protocol understanding model is based on fine-tuning training of a general reasoning model, including:

[0065] Step 201: construct a non-volatile memory host controller interface specification protocol data set, wherein the non-volatile memory host controller interface specification protocol data set includes key information of the non-volatile memory host controller interface specification protocol and key information of a device interaction command line tool;

[0066] Step 202: fine-tune the general reasoning model based on the key information of the non-volatile memory host controller interface specification protocol and the key information of the device interaction command line tool to generate an incremental tuning parameter file;

[0067] Step 203: Load the incremental tuning parameter file into the general reasoning model, and fine-tune some parameters in the general reasoning model.

[0068] In an embodiment of the present invention, the step of constructing a non-volatile memory host controller interface specification protocol data set includes:

[0069] Step 2011: Obtain a non-volatile memory host controller interface specification, and obtain a completion queue entry and a submission queue entry in the non-volatile memory host controller interface specification, wherein the completion queue entry is obtained based on a command execution result, and the submission queue entry is obtained based on a command submission result;

[0070] Step 2012: construct a protocol knowledge graph based on the completion queue entries and submission queue entries;

[0071] Step 2013: Add the key commands and command parameters of the device interactive command line tool to the protocol knowledge graph;

[0072] Step 2014: construct a non-volatile memory host controller interface specification protocol dataset based on the supplemented protocol knowledge graph.

[0073] In the embodiment of the present invention, the non-volatile memory host controller interface specification data set includes all key concepts, attributes, relationships and constraints of the non-volatile memory host controller interface specification protocol.

[0074] In order to ensure that the general large model can deeply understand the operating principle of the non-volatile memory host controller interface specification protocol, an embodiment of the present invention constructs a knowledge graph, which specifically describes the non-volatile memory host controller interface specification protocol standard in detail. The knowledge graph includes the non-volatile memory host controller interface specification protocol and its nvme-cli tool. The nvme-cli tool is a command line tool for interacting with the non-volatile memory host controller interface specification device. The non-volatile memory host controller interface specification device is a high-performance storage and access protocol designed for solid-state drives. By constructing a knowledge graph, the general understanding model can fully understand and analyze the complex characteristics of the non-volatile memory host controller interface specification device protocol.

[0075] In addition, the use of the nvme-cli tool is also incorporated into the construction of the knowledge graph. The subcommands and their parameters in the nvme-cli tool, such as admin-passthru, io-passthru, and show-regs, are the core functions of the nvme-cli tool. By annotating them in detail, the general understanding model can better understand how to use the nvme-cli tool to interact with the non-volatile memory host controller interface specification solid-state drive device.

[0076] Based on any of the above embodiments, the protocol understanding model is based on fine-tuning the general reasoning model and further includes:

[0077] Based on the protocol standard document, typical protocol test cases in each feature module are obtained, and the running process of the non-volatile memory host controller interface specification protocol is described in the format of (passage, question, answer), where passage describes a feature or command of the protocol, question describes a specific question, and answer describes a detailed solution to the question;

[0078] Input (passage, question, answer) into a universal reasoning model so that the universal reasoning model can learn the operation process of the non-volatile memory host controller interface specification protocol and fine-tune some parameters in the universal reasoning model.

[0079] In order to further strengthen the general understanding model's understanding of the non-volatile memory host controller interface specification protocol, the embodiment of the present invention also organizes typical non-volatile memory host controller interface specification protocol test cases in each feature module based on the protocol standard document. For example, for the test case of the Write command, the passage describes the basic functions and parameters of the Write command, the question raises the question of how to verify the Write command using the sgl method, and the answer gives the specific test steps and expected results. In this way, the large model can have a deeper understanding of the operating principle of the non-volatile memory host controller interface specification protocol, and can generate more accurate and comprehensive test cases.

[0080] In an embodiment of the present invention, a general large model, for example, adopts a Chatglm-6b model, and fine-tuning adopts Full Fine-Tuning technology. Training is performed based on a generated non-volatile memory host controller interface specification protocol data set, and some parameters of the Chatglm-6b model are optimized to generate an incremental tuning parameter file to obtain a non-volatile memory host controller interface specification protocol understanding model with deep understanding and reasoning capabilities, thereby achieving fast and low-cost model tuning.

[0081] Based on any of the above embodiments, the step of inputting the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model to generate a test case includes:

[0082] Step 301, traverse the protocol standard document of the non-volatile memory host controller interface specification protocol to be tested, and intercept partial texts one by one;

[0083] Step 302: Identify the type of each partial text, perform structural processing on the partial text, and identify the commands related to the protocol in the partial text;

[0084] Step 303: Fill the command related to the protocol in the partial text into the Prompt template of the corresponding type;

[0085] Step 304: input the filled Prompt template into the protocol understanding model to generate a corresponding test case.

[0086] In an embodiment of the present invention, an artificial intelligence general understanding model is used to load the incremental parameter file generated by tuning, and corresponding test cases are generated according to the designed Prompt template. According to the protocol understanding model, test cases covering various aspects of the NVMe protocol are automatically generated, including commands, parameters, return values, status codes, etc.

[0087] like Figure 2 As shown, in the embodiment of the present invention, the filled Prompt template is input into the protocol understanding model to generate a corresponding test case, including:

[0088] If the type of the partial text is a paragraph, the Prompt template guides the protocol comprehension model to generate a lightweight data exchange format test case including detailed test steps and expected results;

[0089] If the type of the partial text is a form, the Prompt template will guide the protocol understanding model to generate a test case for a specific command parameter bit setting;

[0090] If the type of the partial text is a picture, the Prompt template will guide the protocol comprehension model to generate a test case based on the picture content.

[0091] Based on any of the above embodiments, Figure 3 As shown, the test cases generated in each round are evaluated, including:

[0092] The test case is evaluated for completeness, specifically including outputting the completeness of the test case according to whether the current test case covers the test cases corresponding to the paragraphs, forms and pictures respectively.

[0093] Performing integrity assessment on the test case, for example, including determining whether a read command covers protocol content related to the read command;

[0094] The effectiveness of the test case is evaluated, specifically including outputting the effectiveness of the test case according to whether the command parameter execution result of the device interactive command line tool is correct.

[0095] In the embodiment of the present invention, the evaluation method adopts a method combining expert review and automated testing to improve the accuracy and efficiency of the evaluation. Expert review consists of domain experts manually reviewing the test cases and providing professional opinions and feedback. Automated testing uses specialized test tools and scripts to automatically execute test cases and record the results.

[0096] The reliability of the test case is evaluated, specifically including executing the test case under different test environments and conditions, and verifying the reliability of the output test case according to the test case execution results.

[0097] Based on any of the above embodiments, optimizing the test cases generated in each round includes:

[0098] Step 401: Screen the test cases generated in each round according to the completeness, validity and reliability of the test cases to obtain high-quality test cases;

[0099] Step 402: Optimize the non-volatile memory host controller interface specification protocol data set with the high-quality test case;

[0100] Step 403: Use the optimized non-volatile memory host controller interface specification protocol data set to train the protocol understanding model to obtain an optimized protocol understanding model.

[0101] In the embodiment of the present invention, the newly acquired high-quality test cases are used as model inputs, and this process is repeated to achieve the goal of model optimization.

[0102] Step 404: regenerate test cases according to the optimized protocol understanding model, use the regenerated test cases to test the non-volatile memory host controller interface specification protocol to be tested, and output protocol test results.

[0103] like Figure 4 As shown, an embodiment of the present invention constructs a non-volatile memory host controller interface specification protocol dataset, fine-tunes the large model, automatically generates test cases, evaluates the test cases, and determines whether the quality of the test cases meets expectations. If so, a protocol understanding model is generated; otherwise, the non-volatile memory host controller interface specification protocol dataset is optimized according to the newly generated high-quality test cases.

[0104] In an embodiment of the present invention, the test case includes a case title, pre-conditions, test steps and expected results, and optimizing the test case generated in each round further includes:

[0105] If the test case title, preconditions, test steps and expected results of the newly generated test case are the same as those of the existing test case, the newly generated test case is determined to be a duplicate test case, and the duplicate test case is deleted from the optimized non-volatile memory host controller interface specification protocol data set.

[0106] During the evaluation process, a duplicate test case detection mechanism is introduced to detect whether the newly generated test cases are duplicates of existing test cases. Duplicate cases can be defined based on multiple dimensions such as the title, preconditions, test steps, expected results, etc. If duplicate cases are found, they are automatically excluded from the candidate test case set.

[0107] The logic device interface specification protocol test method provided by the present invention fully utilizes the advantages of the artificial intelligence large model in deep understanding and reasoning, and fine-tunes the general understanding model through the non-volatile memory host controller interface specification protocol specific data set, thereby realizing the automatic generation of non-volatile memory host controller interface specification protocol standard test cases, improving the coverage of the test scenario, and at the same time, ensuring that the generated test cases have high integrity, effectiveness and reliability through test case evaluation. By continuously optimizing and adjusting the protocol model, the continuous improvement of the test cases is achieved, providing strong support for the research and development and testing of NVMe SSD products.

[0108] The following is a description of the logic device interface specification protocol testing device provided by the present invention. The logic device interface specification protocol testing device described below and the logic device interface specification protocol testing method described above can be referenced to each other.

[0109] Figure 5 A schematic diagram of a logic device interface specification protocol test device provided by an embodiment of the present invention, such as Figure 5 As shown, the logic device interface specification protocol testing device provided by the embodiment of the present invention includes:

[0110] An input module 501 is used to obtain a non-volatile memory host controller interface specification protocol to be tested, input the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model, and generate a test case;

[0111] A testing module 502 is used to test the non-volatile memory host controller interface specification protocol to be tested based on the test case and output a protocol test result;

[0112] The protocol understanding model is obtained based on fine-tuning training of a general reasoning model and evaluation and optimization based on test cases generated in each round.

[0113] In an embodiment of the present invention, the protocol understanding model is based on fine-tuning training of a general reasoning model, including:

[0114] Constructing a non-volatile memory host controller interface specification protocol data set, wherein the non-volatile memory host controller interface specification protocol data set includes key information of the non-volatile memory host controller interface specification protocol and key information of a device interaction command line tool;

[0115] Fine-tune the general inference model based on key information of the non-volatile memory host controller interface specification protocol and key information of the device interaction command line tool to generate an incremental tuning parameter file;

[0116] The incremental tuning parameter file is loaded into the general reasoning model, and some parameters in the general reasoning model are fine-tuned.

[0117] In an embodiment of the present invention, the step of constructing a non-volatile memory host controller interface specification protocol data set includes:

[0118] Acquire a non-volatile memory host controller interface specification, and acquire a completion queue entry and a submission queue entry in the non-volatile memory host controller interface specification, wherein the completion queue entry is acquired based on a command execution result, and the submission queue entry is acquired based on a command submission result;

[0119] Building a protocol knowledge graph based on the completion queue entries and submission queue entries;

[0120] Supplement the key commands and command parameters of the device interaction command line tool to the protocol knowledge graph;

[0121] A non-volatile memory host controller interface specification protocol dataset is constructed based on the supplemented protocol knowledge graph.

[0122] In an embodiment of the present invention, the protocol understanding model is based on fine-tuning the general reasoning model and further includes:

[0123] Based on the protocol standard document, typical protocol test cases in each feature module are obtained, and the running process of the non-volatile memory host controller interface specification protocol is described in the format of (passage, question, answer), where passage is used to describe a feature or command of the protocol, question is used to describe a specific question, and answer is used to describe a detailed answer to the question;

[0124] Input (passage, question, answer) into a universal reasoning model so that the universal reasoning model can learn the operation process of the non-volatile memory host controller interface specification protocol and fine-tune some parameters in the universal reasoning model.

[0125] In an embodiment of the present invention, the step of inputting the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model to generate a test case includes:

[0126] Traverse the protocol standard documents of the interface specification protocol of the non-volatile memory host controller to be tested, and extract partial texts one by one;

[0127] Identify the type of each partial text, perform structural processing on the partial text, and identify the commands related to the protocol in the partial text;

[0128] Fill the commands related to the protocol in the partial text into the corresponding type of Prompt template;

[0129] The filled Prompt template is input into the protocol understanding model to generate the corresponding test case.

[0130] In the embodiment of the present invention, the step of inputting the filled Prompt template into the protocol understanding model to generate a corresponding test case includes:

[0131] If the type of the partial text is a paragraph, the Prompt template guides the protocol comprehension model to generate a lightweight data exchange format test case including detailed test steps and expected results;

[0132] If the type of the partial text is a form, the Prompt template will guide the protocol understanding model to generate a test case for a specific command parameter bit setting;

[0133] If the type of the partial text is a picture, the Prompt template will guide the protocol comprehension model to generate a test case based on the picture content.

[0134] In an embodiment of the present invention, the test cases generated in each round are evaluated, including:

[0135] The test case is evaluated for completeness, specifically including outputting the completeness of the test case according to whether the current test case covers the test cases corresponding to the paragraphs, forms and pictures respectively.

[0136] In an embodiment of the present invention, the test cases generated in each round are evaluated, including:

[0137] The effectiveness of the test case is evaluated, specifically including outputting the effectiveness of the test case according to whether the command parameter execution result of the device interactive command line tool is correct.

[0138] In an embodiment of the present invention, the test cases generated in each round are evaluated, including:

[0139] The reliability of the test case is evaluated, specifically including executing the test case under different test environments and conditions, and verifying the reliability of the output test case according to the test case execution results.

[0140] In an embodiment of the present invention, each round of generated test cases is optimized, including:

[0141] Screen the test cases generated in each round according to their completeness, effectiveness, and reliability to obtain high-quality test cases;

[0142] Optimizing the non-volatile memory host controller interface specification protocol data set with the high-quality test cases;

[0143] Using the optimized non-volatile memory host controller interface specification protocol data set to train the protocol understanding model, to obtain an optimized protocol understanding model;

[0144] Regenerate test cases according to the optimized protocol understanding model, use the regenerated test cases to test the non-volatile memory host controller interface specification protocol to be tested, and output protocol test results.

[0145] In an embodiment of the present invention, the test case includes a case title, pre-conditions, test steps and expected results, and optimizing the test case generated in each round further includes:

[0146] If the test case title, preconditions, test steps and expected results of the newly generated test case are the same as those of the existing test case, the newly generated test case is determined to be a duplicate test case, and the duplicate test case is deleted from the optimized non-volatile memory host controller interface specification protocol data set.

[0147] The logic device interface specification protocol testing device provided by the embodiment of the present invention obtains the non-volatile memory host controller interface specification protocol to be tested, inputs the non-volatile memory host controller interface specification protocol to be tested into the protocol understanding model, generates test cases; tests the non-volatile memory host controller interface specification protocol to be tested based on the test cases, and outputs the protocol test results; wherein the protocol understanding model is obtained based on fine-tuning and training the general reasoning model, and is evaluated and optimized according to each round of generated test cases. The content generation technology of artificial intelligence is used to optimize the test of non-volatile memory host controller interface specification solid-state hard disk products. Starting from the non-volatile memory host controller interface specification protocol standard, the protocol understanding model in the field of non-volatile memory host controller interface specification testing is obtained by fine-tuning the general model, and the test cases are generated and optimized based on the protocol understanding model to improve the test efficiency and coverage, thereby improving the quality of non-volatile memory host controller interface specification solid-state hard disk products.

[0148] Figure 6 An example of a server physical structure diagram is shown below: Figure 6 As shown, the server may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The memory 630 includes a computer program, an operating system and the acquired graph structure data, and the processor 610 may call the logic instructions in the memory 630 to execute the logic device interface specification protocol test method, the method comprising: obtaining the interface specification protocol of the non-volatile memory host controller to be tested, inputting the interface specification protocol of the non-volatile memory host controller to be tested into the protocol understanding model, and generating a test case; testing the interface specification protocol of the non-volatile memory host controller to be tested based on the test case, and outputting the protocol test result; wherein the protocol understanding model is obtained based on fine-tuning and training the general reasoning model, and evaluating and optimizing according to the test cases generated in each round.

[0149] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0150] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the logic device interface specification protocol testing method provided by the above-mentioned methods, the method comprising: obtaining the interface specification protocol of a non-volatile memory host controller to be tested, inputting the interface specification protocol of the non-volatile memory host controller to be tested into a protocol understanding model, and generating a test case; testing the interface specification protocol of the non-volatile memory host controller to be tested based on the test case, and outputting the protocol test result; wherein the protocol understanding model is obtained based on fine-tuning training of a general reasoning model, and evaluation and optimization according to the test cases generated in each round.

[0151] On the other hand, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the logic device interface specification protocol testing method described in any one of the above items, the method comprising: obtaining a non-volatile memory host controller interface specification protocol to be tested, inputting the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model, and generating a test case; testing the non-volatile memory host controller interface specification protocol to be tested based on the test case, and outputting a protocol test result; wherein the protocol understanding model is obtained based on fine-tuning training of a general reasoning model, and is evaluated and optimized according to the test cases generated in each round.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for testing a logical device interface specification protocol. It is characterized in that include: Obtaining a non-volatile memory host controller interface specification protocol to be tested, inputting the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model, and generating a test case; Testing the non-volatile memory host controller interface specification protocol to be tested based on the test case, and outputting a protocol test result; The protocol understanding model is obtained by fine-tuning and training the general reasoning model and evaluating and optimizing the test cases generated in each round. Fine-tune the general inference model, including: Constructing a non-volatile memory host controller interface specification protocol data set, wherein the non-volatile memory host controller interface specification protocol data set includes key information of the non-volatile memory host controller interface specification protocol and key information of a device interaction command line tool; Fine-tune the general inference model based on key information of the non-volatile memory host controller interface specification protocol and key information of the device interaction command line tool to generate an incremental tuning parameter file; The incremental tuning parameter file is loaded into the general reasoning model, and some parameters in the general reasoning model are fine-tuned.

2. The method for testing the logical device interface specification protocol according to claim 1, It is characterized in that The method of constructing a non-volatile memory host controller interface specification protocol data set includes: Acquire a non-volatile memory host controller interface specification, and acquire a completion queue entry and a submission queue entry in the non-volatile memory host controller interface specification, wherein the completion queue entry is acquired based on a command execution result, and the submission queue entry is acquired based on a command submission result; Building a protocol knowledge graph based on the completion queue entries and the submission queue entries; Supplement the key commands and command parameters of the device interaction command line tool to the protocol knowledge graph; A non-volatile memory host controller interface specification protocol dataset is constructed based on the supplemented protocol knowledge graph.

3. The method for testing the logical device interface specification protocol according to claim 1, It is characterized in that Fine-tune the general inference model, including: Based on the protocol standard document, typical protocol test cases in each feature module are obtained, and the running process of the non-volatile memory host controller interface specification protocol is described in the format of (passage, question, answer), where passage is used to describe a feature or command of the protocol, question is used to describe a specific question, and answer is used to describe a detailed answer to the question; Input (passage, question, answer) into a universal reasoning model so that the universal reasoning model can learn the operation process of the non-volatile memory host controller interface specification protocol and fine-tune some parameters in the universal reasoning model.

4. The method for testing the logical device interface specification protocol according to claim 1, It is characterized in that The step of inputting the non-volatile memory host controller interface specification protocol to be tested into a protocol understanding model to generate a test case comprises: Traverse the protocol standard documents of the interface specification protocol of the non-volatile memory host controller to be tested, and extract partial texts one by one; Identify the type of each partial text, perform structural processing on the partial text, and identify the commands related to the protocol in the partial text; Fill the commands related to the protocol in the partial text into the corresponding type of Prompt template; The filled Prompt template is input into the protocol understanding model to generate the corresponding test case.

5. The method for testing the logical device interface specification protocol according to claim 4, It is characterized in that The method further comprises: identifying the type of each partial text, performing structural processing on the partial text, and identifying the commands related to the protocol in the partial text; Fill the commands related to the protocol in the partial text into the corresponding type of Prompt template; Input the filled Prompt template into the protocol understanding model to generate corresponding test cases, including: If the type of the text content is a paragraph, the Prompt template guides the protocol understanding model to generate a lightweight data exchange format test case including detailed test steps and expected results; If the type of the text content is a form, the Prompt template will guide the protocol understanding model to generate a test case for a specific command parameter bit setting; If the type of the text content is a picture, the Prompt template will guide the protocol understanding model to generate a test case based on the picture content.

6. The method for testing the logical device interface specification protocol according to claim 1, It is characterized in that The test cases generated in each round are evaluated, including: The test case is evaluated for completeness, specifically including outputting the completeness of the test case according to whether the current test case covers the test cases corresponding to the paragraphs, forms and pictures respectively.

7. The method for testing the logical device interface specification protocol according to claim 1, It is characterized in that The test cases generated in each round are evaluated, including: The effectiveness of the test case is evaluated, specifically including outputting the effectiveness of the test case according to whether the command parameter execution result of the device interactive command line tool is correct.

8. The method for testing the logical device interface specification protocol according to claim 1, It is characterized in that The test cases generated in each round are evaluated, including: The reliability of the test case is evaluated, specifically including executing the test case under different test environments and conditions, and verifying the reliability of the output test case according to the test case execution results.

9. The method for testing the logical device interface specification protocol according to claim 1, It is characterized in that Optimize the test cases generated in each round, including: Screen the test cases generated in each round according to their completeness, effectiveness, and reliability to obtain high-quality test cases; Optimizing the non-volatile memory host controller interface specification protocol data set with the high-quality test cases; Using the optimized non-volatile memory host controller interface specification protocol data set to train the protocol understanding model, to obtain an optimized protocol understanding model; Regenerate test cases according to the optimized protocol understanding model, use the regenerated test cases to test the non-volatile memory host controller interface specification protocol to be tested, and output protocol test results.

10. The method for testing the logical device interface specification protocol according to claim 9, It is characterized in that The test case includes a case title, preconditions, test steps and expected results. Optimizing the test case generated in each round also includes: If the test case title, preconditions, test steps and expected results of the newly generated test case are the same as those of the existing test case, the newly generated test case is determined to be a duplicate test case, and the duplicate test case is deleted from the optimized non-volatile memory host controller interface specification protocol data set.

11. A logical device interface specification protocol test device, It is characterized in that include: An input module, used for obtaining the interface specification protocol of the non-volatile memory host controller to be tested, inputting the interface specification protocol of the non-volatile memory host controller to be tested into a protocol understanding model, and generating a test case; A test module, used for testing the non-volatile memory host controller interface specification protocol to be tested based on the test case, and outputting the protocol test result; The protocol understanding model is obtained by fine-tuning and training the general reasoning model and evaluating and optimizing the test cases generated in each round. Fine-tune the general inference model, including: Constructing a non-volatile memory host controller interface specification protocol data set, wherein the non-volatile memory host controller interface specification protocol data set includes key information of the non-volatile memory host controller interface specification protocol and key information of a device interaction command line tool; Fine-tune the general inference model based on key information of the non-volatile memory host controller interface specification protocol and key information of the device interaction command line tool to generate an incremental tuning parameter file; The incremental tuning parameter file is loaded into the general reasoning model, and some parameters in the general reasoning model are fine-tuned.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the logic device interface specification protocol testing method according to any one of claims 1 to 10 is implemented.

13. A non-transitory readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the logic device interface specification protocol testing method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the logic device interface specification protocol testing method according to any one of claims 1 to 10 is implemented.