Communication protocol conformance test method and system

Abstract

The invention provides a communication protocol consistency test method and system. Key function points are extracted by analyzing a target protocol document; based on the key function points, a predefined first prompt template is applied to generate a test case prompt, a large language model is guided to generate a standardized test case corresponding to each key function point, and the first prompt template comprises k few-sample examples; dynamically retrieving a related code context from a pre-constructed long-term memory library through a retrieval enhancement generation mechanism, generating a code generation prompt by applying a predefined second prompt template based on the standardized test case and the related code context, and generating an executable test code based on the code generation prompt; and performing iterative optimization on the executable test code through the large language model, and generating a new code generation prompt based on a code execution result of the previous iteration and the retrieved related code context in each iteration. The test case can be automatically generated and dynamically adapted, and the test coverage rate, efficiency and flexibility are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication protocol testing, and in particular to a communication protocol consistency testing method and system. BACKGROUND

[0002] Communication protocol consistency testing usually needs to verify whether the protocol implementation strictly conforms to the standard specification. At present, protocol testing mostly relies on test engineers to manually write test case scripts, which not only consumes time and effort, but also is prone to errors.

[0003] In a typical scenario, a tester often needs to spend weeks writing a large number of scripts for verification of different protocol function points. With the increase of protocol complexity and implementation versions, the traditional manual testing is extremely inefficient, costly, and difficult to update test cases in a timely manner to adapt to the differences in protocol implementation of various manufacturers or versions.

[0004] Therefore, there is an urgent need for a solution that can automatically generate and dynamically adapt test cases to improve test coverage, efficiency and flexibility. SUMMARY

[0005] The present application provides a communication protocol consistency testing method and system to solve the defects of low efficiency, high cost and difficulty in updating test cases in a timely manner to adapt to the differences in protocol implementation of various manufacturers or versions in the prior art, and to realize the automatic generation and dynamic adaptation of test cases.

[0006] The present application provides a communication protocol consistency testing method, comprising the following steps: parsing a target protocol document, extracting key function points from the target protocol document based on RFC 2119 keyword rules, the key function points referring to core clauses or behavior description paragraphs with explicit constraint requirements; based on the key function points, applying a predefined first prompt template to generate a test case prompt, guiding a large language model to generate a standardized test case corresponding to each key function point, the first prompt template containing k few-sample examples; dynamically retrieving relevant code context from a pre-constructed long-term memory library through a retrieval enhancement generation mechanism, and based on the standardized test case and the relevant code context, applying a predefined second prompt template to generate a code generation prompt, and based on the code generation prompt, generating executable test code, the long-term memory library being constructed based on the encoding results of the target implementation library source code by an embedding model; iteratively optimizing the executable test code by the large language model, wherein each iteration generates a new code generation prompt based on the code execution result of the previous iteration and the retrieved relevant code context, and the loop is executed until success or the preset iteration upper limit is reached, and the execution result is captured; analyzing the execution result to generate a protocol consistency test report.

[0007] According to the communication protocol consistency test method provided by the application, the target protocol document is parsed, and the key function points are extracted from the target protocol document based on the RFC2119 keyword rule, which specifically includes: detecting whether the target protocol document conforms to the RFC 2119 standard; if the target protocol document conforms to the RFC 2119 standard, extracting a complete paragraph containing a capital keyword through a regular expression; or if the target protocol document does not conform to the RFC 2119 standard, performing case-insensitive keyword extraction and obtaining a complete paragraph containing the keyword.

[0008] According to the communication protocol consistency test method provided by the application, the first prompt template contains: a task description field for defining a generation task, activating an inference process, and specifying an output format; a test case example for providing a conversion example from a protocol function point to a standardized test case for a large language model through few-shot learning.

[0009] According to the communication protocol consistency test method provided by the application, the second prompt template contains: a role definition field for specifying a large language model as a test code generation expert; a task description field for defining a task of the large language model when generating code for the first time and a new task of the large language model in a debugging and optimization stage, and instructing the large language model to perform step-by-step inference during task execution, and requiring the large language model to output test code in a predetermined structured format; a memory field for representing historical information obtained from a short-term memory stack; a context field for representing a latest relevant code segment retrieved from a long-term memory library for a current problem through a retrieval enhancement generation mechanism.

[0010] According to the communication protocol consistency test method provided by the application, the predetermined structured format is a JSON format, which contains a code block list and an execution order identifier.

[0011] According to the communication protocol consistency test method provided by the application, the retrieval enhancement generation mechanism dynamically retrieves relevant code context from a pre-constructed long-term memory library, which includes: when generating executable test code for the first time, the retrieved code context includes relevant functions, class definitions, or call examples; In the iterative optimization phase, the retrieved code context and the code execution result of the previous iteration are jointly used as the basis for generating a new code generation prompt.

[0012] According to the communication protocol consistency test method provided by the application, the executable test code is iteratively optimized by the large language model, and specifically includes the following steps: A short-term memory stack is constructed to cache historical information, including historical code execution results, historical retrieval results, and historical LLM output results. The historical information in the short-term memory stack is fused with the newly retrieved code context to generate a new code generation prompt. Based on the new code generation prompt, an iterative optimization algorithm is used to iteratively optimize the executable test code.

[0013] According to the communication protocol consistency test method provided by the application, the size of the cache window of the short-term memory stack is a dynamic value, satisfying where i is the current iteration round.

[0014] The application also provides a communication protocol consistency test system, including the following modules: A function point extraction module is used to parse a target protocol document, extract key function points from the target protocol document based on RFC 2119 keyword rules, and the key function points refer to core clauses or behavior description paragraphs with explicit constraint requirements. A test case generation module is used to generate test case prompts based on the key function points, apply a predefined first prompt template to guide a large language model to generate standardized test cases corresponding to each key function point, and the first prompt template contains k few-sample examples. A test code generation module is used to dynamically retrieve relevant code context from a pre-constructed long-term memory library through a retrieval enhancement generation mechanism, and based on the standardized test cases and the relevant code context, a second prompt template is applied to generate a code generation prompt, and based on the code generation prompt, executable test code is generated. The long-term memory library is constructed based on the encoding results of the target implementation library source code by an embedding model. An iterative optimizer is used to iteratively optimize the executable test code by the large language model, wherein a new code generation prompt is generated based on the code execution result of the previous iteration and the retrieved relevant code context for each iteration, and the loop is executed until success or the preset iteration upper limit is reached, and the execution result is captured. A report generation module is used to analyze the execution result and generate a protocol consistency test report.

[0015] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the communication protocol consistency test method according to any one of the above when executing the computer program.

[0016] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the communication protocol consistency test method according to any one of the above.

[0017] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the communication protocol consistency test method according to any one of the above.

[0018] The application provides a communication protocol consistency test method and system, which extracts key function points from a target protocol document based on RFC2119 keyword rules by analyzing the target protocol document, wherein the key function points refer to core clauses or behavior description paragraphs with explicit constraint requirements; generates test case prompts by applying a predefined first prompt template based on the key function points, guides a large language model to generate standardized test cases corresponding to each key function point, and the first prompt template contains k few-sample examples; dynamically retrieves relevant code contexts from a pre-constructed long-term memory library through a retrieval enhancement generation mechanism, and generates code generation prompts by applying a predefined second prompt template based on the standardized test cases and the relevant code contexts, generates executable test code based on the code generation prompts, and the long-term memory library is constructed based on the encoding results of target implementation library source code by an embedding model; iteratively optimizes the executable test code by the large language model, wherein a new code generation prompt is generated based on the code execution results of the previous iteration and the retrieved relevant code contexts in each iteration, and the iteration is performed cyclically until success or the preset iteration upper limit is reached, and the execution results are captured; analyzes the execution results to generate a protocol consistency test report. The application automatically completes protocol function point extraction, test case generation, test code generation and execution feedback iterative optimization through natural language understanding and code generation capabilities, forms a complete closed loop from protocol document to consistency test results, converts complex protocol consistency testing into an automated process, greatly reduces manual intervention, and improves test coverage and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0020] Figure 1 is a flowchart of the communication protocol consistency test method provided by the present application.

[0021] Figure 2 is a workflow diagram of the communication protocol consistency test method provided by the present application.

[0022] Figure 3 is a prompt template structure example for test case generation provided by the present application.

[0023] Figure 4 is a code JSON structure template provided by the present application.

[0024] Figure 5 is a prompt template structure example for test code generation and iterative optimization provided by the present application.

[0025] Figure 6 is a code example of the iterative optimization algorithm provided by the present application.

[0026] Figure 7 is a structural diagram of the communication protocol consistency test system provided by the present application.

[0027] Figure 8 is a structural diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0029] The present application will be described in detail below in combination with the drawings in the specification. The specific operation methods in the method embodiments can also be applied to the device embodiments or the system embodiments. In the description of the present application, unless otherwise specified, “at least one” includes one or more. “Multiple” refers to two or more. For example, at least one of A, B and C includes: A alone, B alone, A and B together, A and C together, B and C together, and A, B and C together. In the present application, “ / ” means or, for example, A / B can mean A or B; “and / or” in this document only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone.

[0030] Consistency testing of communication protocols is a method used to verify that a protocol implementation meets the requirements defined in the protocol specification. Its main goal is to ensure that the protocol implementation behaves consistently across different scenarios and input conditions, thus guaranteeing compatibility and interoperability between different protocol implementations. A complete protocol consistency testing process typically includes the following key steps: Specification analysis: In-depth understanding and analysis of the protocol's standard documents to extract well-defined behaviors and requirements.

[0031] Test case design: Developing comprehensive test cases based on the protocol specification.

[0032] Test case implementation: Writing test scripts or code using the protocol implementation to be tested, which specifically implements the test cases.

[0033] Test execution: Running the test case code in a properly configured test environment and collecting results.

[0034] Result analysis and evaluation: Analyzing test results to determine whether the system under test meets the specified requirements.

[0035] Large language models (LLMs) are artificial intelligence models based on deep learning technology, pre-trained on a large amount of text data to capture rich semantic and contextual relationships. Well-known LLMs such as the GPT series, LLaMA series, and DeepSeek series have the ability to generate detailed content, understand complex semantics, and perform a wide range of tasks, including text generation, question answering, and code writing.

[0036] The current consistency testing process has significant limitations, mainly due to its high dependence on manual operations. Consistency testing involves a large number of detailed test cases that need to be manually written, making the process labor-intensive and time-consuming. In addition, implementing these test cases requires a large amount of test code development, further increasing the cost of development and maintenance. These limitations significantly reduce the efficiency and flexibility of consistency testing.

[0037] Traditional protocol testing methods mainly include the following categories: Specification-driven testing (Specification-Driven Testing): This method involves manually analyzing protocol documents to identify all functional points and state behaviors, relying on test engineers to manually design test cases that cover the content of the specification. This is currently the most common method of consistency testing, but it relies heavily on human experience, has low testing efficiency, and is prone to missing edge scenarios.

[0038] Model-Based Testing: Formal modeling methods such as Finite State Machine (FSM) and UML state chart are used to model the state transition and message interaction process of the protocol, and state coverage test cases are automatically generated. Although it improves the degree of automation to a certain extent, the model construction process itself still highly depends on manual modeling work, and the maintenance cost is high, and it is difficult to adapt to frequent changes in protocols.

[0039] Implementation-Based Testing: By analyzing the existing protocol implementation, combining code analysis, coverage analysis, and assertion injection, test cases are designed. This method focuses on verifying the implementation behavior, but is limited by the quality and testability of the code itself, and lacks the ability to verify the consistency of the standard document level.

[0040] Interoperability Testing: This method verifies the compatibility between different manufacturer protocol implementations, such as whether a client can correctly parse the protocol message sent by another server. Although it can expose potential compatibility problems, it has limited support for standard consistency verification and unclear test coverage.

[0041] Fuzz Testing: Automatically generate a large number of variant input messages to try to trigger abnormal behavior or security vulnerabilities in the protocol implementation. This method is efficient, but the test points are not semantically targeted, making it difficult to systematically verify the correct implementation of the protocol specification. There is currently a tool called LLMIF based on LLM for Internet of Things fuzz testing.

[0042] The above traditional methods are widely used in engineering practice, but they have the following limitations: Highly dependent on human experience, time-consuming and laborious in test design and execution process; Difficult to extend, poor adaptation to multiple versions of protocols and heterogeneous implementations; Low degree of automation, test script generation and update cannot be synchronized with specification changes; Lack of closed-loop mechanism, unable to dynamically repair generated defect test cases during testing process.

[0043] The method using LLM, such as LLMIF, is specific to a particular testing task, and there is no LLM-based method for communication protocol consistency testing. The present invention aims to develop an agent for communication protocol consistency testing based on LLM, which is the first attempt to use the powerful text understanding and code generation capabilities of LLM to automatically complete the consistency testing task of communication protocols.

[0044] Currently, there is no end-to-end consistency testing tool that can seamlessly integrate protocol documents, implementation, and test results. The excellent text understanding and code generation capabilities of LLM provide new possibilities for addressing these shortcomings. By leveraging the powerful language understanding and generation capabilities of LLM, test case design and test code development can be automated or semi-automated, greatly reducing manual effort. Therefore, the present invention aims to integrate LLM into the consistency testing process of communication protocols. The present invention proposes an end-to-end consistency testing framework to improve the efficiency and effectiveness of consistency testing.

[0045] In some embodiments of the present invention, as shown in Figure 1 The present invention provides a method for testing the consistency of a communication protocol, comprising: Step 100, parsing the target protocol document, extracting key functional points from the target protocol document based on the RFC 2119 keyword rules, the key functional points referring to core clauses or behavior description paragraphs with explicit constraint requirements; Step 200, based on the key functional points, applying a pre-defined first prompt template to generate test case prompts, guiding the large language model to generate standardized test cases corresponding to each key functional point, the first prompt template containing k few-shot examples; Step 300, dynamically retrieving relevant code context from a pre-constructed long-term memory library through a retrieval enhancement generation mechanism, and based on the standardized test cases and relevant code context, applying a pre-defined second prompt template to generate code generation prompts, and based on the code generation prompts, generating executable test code, the long-term memory library being constructed based on the encoding results of the target implementation library source code by an embedding model; Step 400, iterative optimization of the executable test code by the large language model, wherein each iteration generates a new code generation prompt based on the code execution results of the previous iteration and the retrieved relevant code context, and the loop is executed until success or the preset iteration upper limit is reached, and the execution results are captured; Step 500, analyzing the execution results, and generating a protocol consistency testing report.

[0046] It should be noted that existing communication protocol consistency testing solutions are highly dependent on human labor, and test case design and script writing need to be completed manually by engineers, which is time-consuming and prone to errors. The dynamic adaptation is poor, making it difficult to cope with multiple versions of protocols / heterogeneous implementation changes, and the test case updates are lagging behind. The closed-loop mechanism is missing, and it is not possible to dynamically repair defect cases based on test results.

[0047] Therefore, the application is based on the automatic function point extraction of the protocol keyword and the few-sample LLM guided test case generation, the retrieval enhancement generation mechanism, the generation of the context code in combination with the protocol implementation library, the iterative code optimization based on the test feedback and the context perception, and the iterative optimization of the executable test code.

[0048] In some possible embodiments of the application, the target protocol document is parsed, and a key function point is extracted from the target protocol document based on the RFC 2119 keyword rule, specifically including: detecting whether the target protocol document meets the RFC 2119 standard; if the target protocol document meets the RFC 2119 standard, extracting a complete paragraph containing a capital keyword through a regular expression; or if the target protocol document does not meet the RFC 2119 standard, performing case-insensitive keyword extraction and obtaining a complete paragraph containing the keyword.

[0049] Specifically, the embodiment provides an implementation of extracting a key function point from the target protocol document. Different key function points are adopted according to whether the target protocol document meets the RFC 2119 standard. The RFC 2119 document extraction accuracy is high, the non-standard document degradation guarantee coverage is high, and the case-insensitive mode is used to deal with non-standard documents.

[0050] In some possible embodiments of the application, the first prompt template includes: a system prompt word for defining a generation task, activating an inference process, and forcing an output format; a test case example for providing a conversion example from a protocol function point to a standardized test case for a large language model through few-sample learning.

[0051] Specifically, the embodiment provides an implementation of a first template for generating a test case prompt. The LLM outputs a structured case (step / input / output), which can greatly improve the efficiency compared with manual review, and focuses on the protocol test scene through role definition. In some possible embodiments of the application, the second prompt template includes: a role definition field for specifying a large language model as a test code generation expert; a task description field for defining a task of the large language model when generating code for the first time and a new task of the large language model in the debugging and optimization stage, and instructing the large language model to perform step-by-step inference during task execution, and forcing the large language model to output test code in a predetermined structured format; a memory field for representing historical information obtained from a short-term memory stack; Context field, used to represent the latest relevant code snippet retrieved from the long-term memory library for the current problem by the retrieval enhancement generation mechanism.

[0052] Specifically, the embodiment provides an implementation of a second template for testing code generation prompts.

[0053] In a possible embodiment, the predetermined structured format is a JSON format, containing a code block list and an execution order identifier.

[0054] Specifically, the embodiment provides an implementation of executable test code, and a JSON parser supports multi-language operation, and code blocks and execution orders ensure correct reorganization of complex test processes. In some possible embodiments of the application, the relevant code context is dynamically retrieved from a pre-constructed long-term memory library through a retrieval enhancement generation mechanism, specifically including: When the executable test code is generated for the first time, the retrieved code context includes relevant functions, class definitions, or call examples; In the iterative optimization phase, the retrieved code context and the code execution result of the previous iteration are jointly used as a basis for generating new code generation prompts.

[0055] Specifically, the embodiment provides an implementation of dynamically retrieving relevant code context from a long-term memory library, through a two-stage RAG mechanism, the initial code pass rate is significantly improved by applying the RAG mechanism in the starting phase, and the debugging success rate is greatly improved by applying the RAG mechanism in the iteration phase. The code execution state returned by the execution module is used to locate the specific repair code, and the debugging manpower is reduced.

[0056] In some possible embodiments of the application, the executable test code is iteratively optimized by the large language model, specifically including: A short-term memory stack is constructed to cache historical information, and the historical information includes historical code execution results, historical retrieval results, and historical LLM output results; The historical information in the short-term memory stack and the newly retrieved code context are fused to generate new code generation prompts; Based on the new code generation prompts, an iterative optimization algorithm is used to iteratively optimize the executable test code.

[0057] In some possible embodiments of the application, the cache window size of the short-term memory stack is a dynamic value, satisfying where i is the current iteration round.

[0058] Specifically, the embodiment provides an implementation of iterative optimization of the executable test code, a dynamic memory window (m belongs to [1, i+1]) simulates the accumulation of engineers' experience, a plurality of rounds of iteration solves most of the code defects, model fine-tuning is driven through a historical error library, error solutions are deposited as a knowledge base, and the continuous learning ability of the model is enhanced.

[0059] In some possible embodiments of the present application, the protocol consistency test report comprises: a functional point compliance determination result; an error type classification of a failed test case; an associated test code segment.

[0060] Specifically, the embodiment provides an implementation of a protocol consistency test report, which accurately locates problems through compliance determination, error classification and associated code. The associated code segment is mapped to a specific functional point, and traceability is achieved.

[0061] In some possible embodiments of the present application, after the standardized test cases are generated, the following is further included: Through a preset abnormal test case filtering interface, user input is received and invalid test cases are removed.

[0062] Specifically, the embodiment provides an implementation of abnormal case filtering, which quickly eliminates edge error cases generated by the LLM through human-computer collaborative quality inspection, and the effectiveness of the filtered test cases is obviously improved.

[0063] The present application provides a communication protocol consistency automatic test method and system based on a large language model (LLM), which automatically completes protocol functional point extraction, test case generation, test code generation and execution feedback iterative optimization through natural language understanding and code generation capabilities, forming a complete closed loop from protocol documents to consistency test results.

[0064] In some possible embodiments of the present application, the communication protocol comprises one or more of CoAP, MQTT and RSocket.

[0065] Specifically, the communication protocol consistency automatic test method based on a large language model provided by the embodiment of the present application is suitable for consistency verification tasks of various network communication protocols (such as CoAP, MQTT and RSocket), and can be widely applied to implementation tests of lightweight communication protocols in the fields of Internet of Things, edge computing and intelligent terminals.

[0066] In some possible embodiments of the present application, the large language model is a pre-training model based on a Transformer architecture, comprising one of a GPT series, an LLaMA series or a DeepSeek series.

[0067] In some embodiments of the present application, as shown in Figure 2 , Figure 2 The workflow of the communication protocol conformance testing method in the embodiment of the present application is shown in the figure. Taking the RSocket protocol as an example, it is assumed that a user wants to verify whether the Python implementation rsocket-py conforms to its specification. The present application first extracts key function points from the protocol document and automatically generates standardized test cases using the LLM generator. Filters can be applied to remove abnormal test cases. For each test case, the present application generates executable test code using the target implementation library. It retrieves the relevant context from the implementation library and integrates this context into the detailed prompt to clearly define the role and task target of the LLM. To ensure executability, the generated code is verified; if a problem is detected, the present application initiates iterative optimization using historical context and error information to dynamically adjust the prompt to debug the code. Once the executable code is generated or the retry limit (6 times) is reached, the present application summarizes the final debugging report and evaluates the test results to determine whether the protocol conformance requirements are met.

[0068] In some embodiments of the present application, the communication protocol conformance testing method provided by the embodiment of the present application can be implemented through the following key modules: a test case generation module, an inference module, a memory module, an execution module, and a summary module.

[0069] For test case generation, the present application designs a new specification-based test case generation method and integrates it into the test case generation module. This method combines heuristic rules and the idea of generating datasets using LLM. Specifically, the method includes the following steps: First, it is detected whether the document follows the RFC 2119 specification. If it does, regular expressions are used to extract paragraphs containing capitalized keywords. Each extracted paragraph is a complete natural paragraph to preserve as much contextual semantic information as possible. We define these paragraphs as function points. If the document does not conform to RFC 2119, the method will default to case-insensitive keyword extraction and repeat the above process.

[0070] After extracting the function points, the LLM is called to generate test cases based on these function points. To ensure that the generated test cases are standardized in format, we introduce a few-shot context learning technique into the LLM. Specifically, the present embodiment constructs a prompt p containing input-output examples , where represents a function point, and is an artificially prepared test case standardized in format. During inference, the test function point The prompt is attached to the LLM, which learns the structure from the provided examples and generates output in the same format . The standardized prompt template is shown in Figure 3 .

[0071] Specifically, as shown in Figure 3 , the system prompt words such as the opening instruction paragraph ("I will give you...") are the "commander-in-chief" of the entire process, responsible for completing the following key tasks: First, define the task boundary (Task Definition): Clearly tell the LLM the specific task to be completed: "Output a test case to verify this new feature point", which avoids irrelevant text generation by the LLM and focuses its thinking on test case creation.

[0072] Second, specify the reasoning process (Reasoning Process): The "think step by-step" instruction triggers the Chain-of-Thought (CoT) reasoning ability of the large language model LLM, requiring it to not directly give the answer, but to reason logically like a human expert, thereby significantly improving the logicality and accuracy of the generated test case.

[0073] Third, control the output format (Output Format Control): Strictly stipulate that the output must be wrapped in the special delimiters << and >>. This has two purposes: Technical parsability: makes it easy for subsequent programs to accurately identify and extract test case content from the LLM's large amount of text output.

[0074] Psychological format reinforcement: again emphasizes to the LLM that it needs to generate a structured "object", rather than casual conversation text.

[0075] Finally, provide the learning strategy (Learning Strategy): "Note that you should pay attention to learn the organization structure of the test case examples", this sentence directly instructs the LLM to imitate the format, style, and detail of the examples provided later ( , …), which is the key instruction to achieve standardized output.

[0076] In general, the system prompt not only describes the test case generation specification, but also defines the framework of the entire task. As shown in Figure 3 indicates, the explicit instruction that the LLM needs to "think step by step" activates the model's chain-of-thought reasoning ability, which is crucial for generating logically rigorous test cases. Moreover, the prompt template contains strict output format instructions (such as "start with <<< and end with >>>"), which achieve technical parsability, allowing the generated test cases to be automatically recognized and extracted by subsequent programs. Finally, the learning strategy ("pay attention to the organizational structure of the examples") is explicitly given, directly instructing the model to imitate the format and style of subsequent examples.

[0077] Therefore, the system prompt instructs the large language model to perform step-by-step reasoning and generate output strictly according to the pre-defined format template (such as using specific delimiters).

[0078] In possible embodiments, to ensure that the generated test cases are both domain-related and effective, a filter can be introduced to allow users to view the function points and their corresponding generated test cases, and to remove any abnormal test cases.

[0079] For code-based prompts in the reasoning module, the reasoning module can construct multi-layer prompts (including startup phase prompts and iteration phase prompts) based on the structured test cases described above, and generate executable test scripts in combination with the code implementation library. The specific process is as follows: Step one, use the embedding model to encode the source code of the protocol implementation library and store it in the long-term memory library; Step two, construct a prompt structure containing roles, tasks, context, additional instructions (including Zero-shot CoT prompts, additional guiding sentences, and output format) for the test protocol and its implementation, as shown in Figure 4 ; Figure 5 ; Specifically, refer to Figure 5Role (role) defines the identity and domain expert role (intelligent reasoning engine, Python developer, network simulation expert) that LLM plays in the dialogue, setting its behavior pattern. Task in StartupStage (startup stage task) defines the task of LLM when generating code for the first time: converting natural language description of test cases into JSON format Python code blocks. Task in Iteration Stage (iteration stage task) defines the new task of LLM in the debugging optimization stage: modifying the code according to the error result after code execution. Instruction (instruction) provides the core operation instruction for LLM. Let's think step by step is a classic thinking chain prompt, instructing the model to reason step by step and avoid errors. Only generate... is an output format mandatory instruction, ensuring that the output is a JSON that can be parsed by the program, not natural language. Memory (memory) represents the history information from the short-term memory stack. represents the multi-round interaction history between the initial input (test case) and the output of the last round of LLM . Context (context) represents the latest relevant code snippets retrieved from the long-term memory library for the current problem through the Retrieve-Augment-Generate (RAG) mechanism . Input (input) represents the specific driving information for this round of iteration. In the startup stage, the input is (test case). In the iteration stage, the input is (execution result or error information of the last round of code).

[0080] Figure 5 This prompt template is extremely clear in structure and finely designed. By precisely defining the role and task to control the behavior pattern of LLM; using thinking chain and mandatory format instruction to improve the quality and parseability of code generation; dynamically integrating memory (history), context (retrieved knowledge), and input (current problem) to provide the most comprehensive and relevant information for LLM. Make LLM not only generate code, but also like a real developer to perform multiple iterations and debugging.

[0081] Step three, in the startup stage, apply the RAG mechanism to retrieve relevant functions, class definitions, or call examples from the long-term memory library as prompt context; in the iteration stage, the context includes not only the relevant content retrieved by RAG, but also the code execution status returned by the execution module (success or error reason); Step four, prompt structure guides LLM to output JSON format test script, including multiple code segments and their execution order.

[0082] For code generation of iterative optimization algorithm, in order to improve the executability and normativity of generated code, the module adopts iterative optimization algorithm to verify and optimize the generated script for no less than 6 rounds until the code that can be successfully executed is generated, and the algorithm pseudo code is as shown in Figure 6 .

[0083] In each iteration process, the input and output of the LLM of the reasoning module are as follows: . (1), wherein, and is a prompt template for guiding the LLM to debug, and the parameters represent the short-term memory cache window size, the historical code execution result , the historical RAG retrieval result and the historical LLM output result are all cached in the short-term memory stack. The RAG retrieval result each time is dynamically called from the retriever in the long-term memory library, that is: . (2).

[0084] Further, for consistency report generation, when the test code passes the verification and is successfully executed, the system analyzes the output result and generates a consistency verification report. The report content includes: whether the current function point conforms to the protocol specification; if the test is successful, the test case and the corresponding test code are given; if the test fails, the failure type is summarized.

[0085] At the same time, the system summarizes the test execution process in a compressed manner, extracts the general experience and saves it in the long-term memory library for future tasks to quickly call and continuously optimize the test capability.

[0086] The above setting of the embodiment of the application realizes the complete automatic protocol test link of automatically extracting test function points from protocol documents, generating test cases and test codes, executing and repairing test processes, and finally outputting a consistency report.

[0087] The communication protocol consistency test method provided by the application extracts test points automatically based on protocol document keywords: by analyzing protocol specification text, automatically identifying function points and constructing test requirements, without manually presetting test scenarios. The method defines function points according to keywords and context in protocol RFC documents, and generates formatted test cases through a small number of example guidance LLM. LLM generates executable code in combination with protocol code library: a test code generation mechanism based on code retrieval enhancement is proposed. The system calls relevant functions or examples in the protocol implementation library as the retrieval source to guide LLM to automatically generate compilable and executable test scripts. This technology uses the code knowledge in the existing protocol implementation (such as CoAP library) to improve the usability and correctness of the generated code. An execution feedback-driven self-iterative correction mechanism: by automatically executing the generated test code and collecting the running results, the execution feedback is input into LLM to iteratively optimize the code. In specific implementation, the test code, execution sequence and feedback information are organized into JSON format through text interaction to guide LLM to generate a revised version of the code according to the error information and new retrieval content. This self-correction mechanism allows the system to perform multiple rounds of debugging like a human tester, improves the final correctness of the generated code, and forms the overall design framework of the system based on LLM for end-to-end communication protocol consistency testing.

[0088] The communication protocol consistency test system provided by the application is described below, and the communication protocol consistency test system described below can be correspondingly referred to the communication protocol consistency test method described above.

[0089] In some specific embodiments of the application, as shown in Figure 7 The scheme provides a communication protocol consistency test system, which comprises: A function point extraction module 71 is configured to analyze a target protocol document, and extract key function points from the target protocol document based on RFC 2119 keyword rules, wherein the key function points refer to core clauses or behavior description paragraphs with explicit constraint requirements. A test case generation module 72 is configured to generate test case prompts based on the key function points by applying a predefined first prompt template, guide a large language model to generate standardized test cases corresponding to each of the key function points, and the first prompt template comprises k few-sample examples. A test code generation module 73 is configured to dynamically retrieve relevant code context from a pre-constructed long-term memory library through a retrieval enhancement generation mechanism, generate code generation prompts based on the standardized test cases and the relevant code context by applying a predefined second prompt template, and generate executable test code based on the code generation prompts, wherein the long-term memory library is constructed based on the encoding results of the target implementation library source code by an embedding model. an iterative optimizer 74 configured to iteratively optimize the executable test code by the large language model, wherein each iteration is based on the code execution result of the previous iteration and the retrieved relevant code context to generate a new code generation prompt, and the iteration is repeated until success or a preset iteration upper limit is reached, and the execution result is captured; a report generation module 75 configured to analyze the execution result and generate a protocol conformance test report.

[0090] It is worth noting that the existing communication protocol conformance test scheme is manually scripted by test personnel according to protocol documents and experience. This way is high in cost and low in efficiency, similar to existing manual testing problems, and is difficult to quickly respond to protocol changes. Or use static analysis tools, use static code analysis or model checking techniques to detect potential defects in protocol implementation, such tools can analyze code defects, but cannot directly generate dynamic test cases that cover all protocol functions, and have limited conformance verification capabilities for protocol standards. Or use a protocol fuzzing engine, apply random or regular fuzzing tools (such as Peach, BFF, etc.) to automatically send variant messages for testing. Although the degree of automation is high, fuzz testing mainly looks for security vulnerabilities, and is weak in targeted testing of protocol features, and requires a lot of time to obtain effective results. Or use a pure LLM generation method, directly let the large model generate test code according to the protocol document, without combining the retrieval and feedback mechanism, this method is difficult to ensure the execution correctness and specification matching degree of the generated code, and is prone to vulnerabilities or logical errors.

[0091] Compared with the above-mentioned schemes, the present application has obvious advantages in test automation and effect, generates test scenarios through keywords, improves the pertinence of test requirements, retrieves and enhances the code library, makes the generated test code directly use existing implementation functions, improves the code accuracy, and further enhances the code quality through iterative correction of execution feedback. Overall, the present application converts complex protocol conformance testing into an automated process, greatly reduces manual intervention, and improves test coverage and efficiency.

[0092] Figure 8 An example of an entity structure diagram of an electronic device is shown in Figure 8As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete mutual communication through the communications bus 840. The processor 810 can invoke a logical instruction in the memory 830 to execute a communication protocol consistency test method, which includes: parsing a target protocol document, extracting a key function point from the target protocol document based on an RFC 2119 keyword rule, the key function point indicating a core clause or behavior description paragraph with an explicit constraint requirement; based on the key function point, applying a predefined first prompt template to generate a test case prompt, guiding a large language model to generate a standardized test case corresponding to each key function point, the first prompt template containing k few-sample examples; dynamically retrieving relevant code context from a pre-constructed long-term memory library through a retrieval enhancement generation mechanism, and based on the standardized test case and the relevant code context, applying a predefined second prompt template to generate a code generation prompt, based on the code generation prompt, generating executable test code, the long-term memory library being constructed based on an encoding result of a target implementation library source code by an embedding model; iteratively optimizing the executable test code by the large language model, wherein each iteration generates a new code generation prompt based on the code execution result of the previous iteration and the retrieved relevant code context, and the loop is executed until success or the preset iteration upper limit is reached, and the execution result is captured; analyzing the execution result to generate a protocol consistency test report.

[0093] In addition, the logical instructions in the memory 830 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0094] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the communication protocol conformance testing method provided by the above-mentioned methods, which comprises: parsing a target protocol document, extracting key function points from the target protocol document based on the RFC 2119 keyword rule, the key function points referring to core clauses or behavior description paragraphs with explicit constraint requirements; based on the key function points, applying a predefined first prompt template to generate a test case prompt, guiding a large language model to generate a standardized test case corresponding to each of the key function points, the first prompt template containing k few-shot examples; through a retrieval-enhanced generation mechanism, dynamically retrieving relevant code contexts from a pre-constructed long-term memory library, and based on the standardized test case and the relevant code contexts, applying a predefined second prompt template to generate a code generation prompt, and based on the code generation prompt, generating executable test code, the long-term memory library being constructed based on the encoding results of target implementation library source code by an embedding model; through the large language model, iteratively optimizing the executable test code, wherein each iteration generates a new code generation prompt based on the code execution results of the previous iteration and the retrieved relevant code contexts, and the loop is executed until success or the preset iteration upper limit is reached, and the execution results are captured; analyzing the execution results to generate a protocol conformance testing report.

[0095] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a communication protocol conformance testing method provided by the above-mentioned methods, which comprises: parsing a target protocol document, extracting key function points from the target protocol document based on the RFC 2119 keyword rule, the key function points referring to core clauses or behavior description paragraphs with explicit constraint requirements; based on the key function points, applying a predefined first prompt template to generate a test case prompt, guiding a large language model to generate a standardized test case corresponding to each of the key function points, the first prompt template containing k few-shot examples; through a retrieval-enhanced generation mechanism, dynamically retrieving relevant code contexts from a pre-constructed long-term memory library, and based on the standardized test case and the relevant code contexts, applying a predefined second prompt template to generate a code generation prompt, and based on the code generation prompt, generating executable test code, the long-term memory library being constructed based on the encoding results of target implementation library source code by an embedding model; through the large language model, iteratively optimizing the executable test code, wherein each iteration generates a new code generation prompt based on the code execution results of the previous iteration and the retrieved relevant code contexts, and the loop is executed until success or the preset iteration upper limit is reached, and the execution results are captured; analyzing the execution results to generate a protocol conformance testing report.

[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0097] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0098] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for testing the consistency of a communication protocol, characterized in that, include: The target protocol document is parsed, and key functional points are extracted from the target protocol document based on the RFC 2119 keyword rules. The key functional points refer to core clauses or behavioral description paragraphs with clear binding requirements. Based on the key functional points, a predefined first prompt template is applied to generate test case prompts, which guides the large language model to generate standardized test cases corresponding to each key functional point. The first prompt template contains k few sample examples. The retrieval enhancement generation mechanism dynamically retrieves relevant code context from the pre-built long-term memory, and generates code generation prompts by applying a predefined second prompt template based on the standardized test cases and relevant code context. Based on the code generation prompts, executable test code is generated. The long-term memory is constructed based on the encoding results of the target implementation library source code using the embedding model. The executable test code is iteratively optimized using the large language model. In each iteration, new code generation prompts are generated based on the code execution result of the previous iteration and the retrieved relevant code context. The process is repeated until success or a preset iteration limit is reached, and the execution result is captured. Analyze the execution results and generate a protocol consistency test report.

2. The communication protocol conformance testing method according to claim 1, characterized in that, The parsing of the target protocol document involves extracting key functional points from the document based on RFC 2119 keyword rules, specifically including: Check whether the target protocol document conforms to the RFC 2119 standard; If the target protocol document conforms to the RFC 2119 standard, extract the complete paragraph containing uppercase keywords using regular expressions; or If the target protocol document does not conform to the RFC 2119 standard, perform case-insensitive keyword extraction and obtain the complete paragraph containing the keywords.

3. The communication protocol conformance testing method according to claim 1, characterized in that, The first prompt template includes: The task description field is used to define the generated task, activate the inference process, and enforce the output format. The test case examples provide paradigms for transforming protocol function points into standardized test cases for large language models through learning from a small number of samples.

4. The communication protocol conformance testing method according to claim 1, characterized in that, The second prompt template includes: The role definition field is used to specify the large language model as the test code generation expert; The task description field is used to define the tasks of the large language model when the code is first generated, as well as the new tasks of the large language model during the debugging and optimization phase. It instructs the large language model to perform step-by-step reasoning during task execution and forces the large language model to output test code in a predetermined structured format; The memory field is used to represent historical information retrieved from the short-term memory stack; The context field is used to represent the latest relevant code snippets retrieved from the long memory for the current problem through the retrieval enhancement generation mechanism.

5. The communication protocol conformance testing method according to claim 4, characterized in that, The predetermined structured format is JSON, which includes a list of code blocks and execution order identifiers.

6. The communication protocol consistency testing method according to claim 1, characterized in that, The method of dynamically retrieving relevant code context from a pre-built long-term memory through a retrieval enhancement generation mechanism includes: When generating executable test code for the first time, the retrieved code context includes relevant functions, class definitions, or call examples; During the iterative optimization phase, the retrieved code context and the code execution results of the previous iteration are used together as the basis for generating new code generation suggestions.

7. The communication protocol conformance testing method according to claim 1, characterized in that, The iterative optimization of the executable test code using the large language model specifically includes: A short-term memory stack is constructed to cache historical information, including historical code execution results, historical retrieval results, and historical LLM output results. By combining the historical information in the short-term memory stack with the newly retrieved code context, new code generation suggestions are generated. Based on the new code generation hints, an iterative optimization algorithm is used to iteratively optimize the executable test code.

8. The communication protocol conformance testing method according to claim 7, characterized in that, The cache window size of the short-term memory stack For dynamic values, satisfying , where i is the current iteration round.

9. A communication protocol conformance testing system, characterized in that, include: The function point extraction module is used to parse the target protocol document and extract key function points from the target protocol document based on the RFC 2119 keyword rules. The key function points refer to core clauses or behavioral description paragraphs with clear binding requirements. The test case generation module is used to generate test case prompts based on the key functional points and apply a predefined first prompt template to guide the large language model to generate standardized test cases corresponding to each key functional point. The first prompt template contains k few sample examples. The test code generation module is used to dynamically retrieve relevant code context from a pre-built long-term memory through a retrieval enhancement generation mechanism, and generate code generation prompts by applying a predefined second prompt template based on the standardized test cases and relevant code contexts. Based on the code generation prompts, executable test code is generated. The long-term memory is constructed based on the encoding results of the target implementation library source code using an embedding model. An iterative optimizer is used to iteratively optimize the executable test code using the large language model. In each iteration, a new code generation suggestion is generated based on the code execution result of the previous iteration and the retrieved relevant code context. The process is repeated until success or a preset iteration limit is reached, and the execution result is captured. The report generation module is used to analyze the execution results and generate a protocol consistency test report.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the communication protocol conformance testing method as described in any one of claims 1 to 8.

Images