A method for automatic extraction of network protocol specifications based on large language models

Abstract

The present invention relates to a method for automatically extracting network protocol specifications based on a large language model. First, each chapter in an RFC document corresponding to a network protocol message format content and a finite state machine content is determined in a matching manner. Then, a target large language model is applied to perform secondary confirmation on the correspondence between each chapter regarding the network protocol, and corresponding extraction fields are analyzed to determine each target chapter. Finally, the target large language model is applied to extract each extraction field in each target chapter, and the combination constitutes a message format and a finite state machine of the network protocol in the RFC document. The design method overcomes the limitations of the prior art, supports the automated understanding and analysis of the network protocol, and realizes more accurate and efficient network protocol extraction.

Description

Technical Field

[0001] The invention relates to a method for automatically extracting network protocol specifications based on a large language model, and belongs to the technical field of network protocol extraction. Background Art

[0002] As the basis for communication between devices and systems, the complexity and diversity of network protocols make manual analysis and implementation time-consuming and error-prone. Request for Comments (RFC) documents are a standardized way to describe network protocols, which define the message structure, state transitions, and communication rules of the protocol in detail. However, RFC documents usually contain a large amount of natural language text, which makes it a challenge to automatically extract a formal model of the protocol from RFCs.

[0003] Traditional natural language processing (NLP) techniques and deep learning-based semantic parsing methods face many limitations when processing RFC documents. First, the protocol definitions in RFCs have the inherent ambiguity of natural language; second, a standard FSM specification is not only based on the information contained in the RFC, but also requires the input of domain experts. In addition, deep learning models require a large amount of high-quality and sufficient annotated data, and in the technical field of network protocols, generating such annotated data is a difficult and expensive task; finally, many key elements in RFCs (such as data flow programs, connection programs, message structures, packet headers, variable definitions, etc.) often appear in the form of text diagrams, which contain information that is not explicitly mentioned in the text.

[0004] Large language models (LLMs) have made breakthrough progress in the field of NLP. Through large-scale pre-training and diverse text data, they have demonstrated the ability to capture complex language patterns and understand complex semantic relationships. LLMs, such as the GPT series of models, are considered to have potential in professional fields such as medical and legal text understanding due to their success in applications such as chatbots, content creation, translation, and code generation. Given the generation capabilities of LLMs and the challenges in extracting protocol information, researchers have begun to explore the application of LLMs in automatically understanding RFCs.

[0005] Although LLMs show potential in processing RFC documents, extracting the precise message format and FSM remains a challenge, requiring a deep understanding of the text graphs in RFC documents and extracting complex structured information from them. In addition, effective prompts need to be designed to guide LLMs to accurately perform the extraction task, while also dealing with the hallucination problem that LLMs may produce, i.e., the model may generate information that is not mentioned in the document. Therefore, using LLMs to extract message formats and FSMs from RFC documents is a challenging but promising area.

[0006] Traditional protocol specification extraction methods mainly rely on manual reading and understanding of RFC documents, which is not only time-consuming and labor-intensive, but also easily affected by subjective understanding, resulting in inconsistency and inaccuracy in the extraction results. In addition, with the increase in the number and complexity of RFC documents, the efficiency and scalability of manual extraction methods are severely limited.

[0007] Although automated extraction techniques such as natural language processing and machine learning models have improved extraction efficiency to a certain extent, these techniques usually require a large amount of annotated data and complex preprocessing. In the context of RFC documents, obtaining high-quality annotated data is particularly difficult because the knowledge and time of protocol experts are limited. In addition, these techniques often need to be customized and adjusted for specific protocols or document formats and lack generalization capabilities.

[0008] Semantic parsing methods based on deep learning face multiple challenges when processing RFC documents. First, the natural language descriptions in RFC documents may be ambiguous, making it difficult for the model to accurately understand the specific requirements of the protocol. Second, deep learning models usually require a lot of computing resources, which is a considerable burden for research and development. Finally, deep learning models have poor interpretability, which is a significant disadvantage in the field of security-sensitive protocol analysis.

[0009] Large language models perform well in processing natural language text, but they still face some challenges when applied to RFC documents. First, LLMs usually rely on a large amount of pre-training data, which may lack sufficient protocol-specific information; in addition, LLMs have limited ability to parse non-text elements in RFC documents (such as diagrams and flow charts), which usually carry key information of the protocol; finally, LLMs may also produce hallucinations, that is, generate content that is not mentioned in the document, which is unacceptable for ensuring the accuracy of the protocol specification.

[0010] Using a large model to extract message formats and FSMs from RFC documents requires addressing the following specific challenges:

[0011] The hallucination problem may cause the message formats and FSM extracted from RFC documents to contain erroneous or fictitious elements, which will directly affect the accuracy of protocol analysis.

[0012] There is a close relationship between message formats and finite state machines. Specifically, message formats act as the "carrier" of communication, while FSM is responsible for controlling how these messages drive the behavior and state transitions of the protocol. Therefore, when extracting the two, it is particularly important to ensure that the results of extracting the two by a large language model remain consistent. This consistency not only affects the accuracy of the extraction results, but also affects the model's correct grasp of the protocol logic when understanding and generating information. Summary of the invention

[0013] The technical problem to be solved by the present invention is to provide a method for automatically extracting network protocol specifications based on a large language model, overcome the limitations of the prior art, support automated understanding and analysis of network protocols, and achieve more accurate and efficient network protocol extraction.

[0014] In order to solve the above technical problems, the present invention adopts the following technical solutions: the present invention designs a method for automatically extracting network protocol specifications based on a large language model, and performs the following steps to extract the message format and finite state machine of the network protocol in the RFC document;

[0015] Step A. According to the preset network protocol specification keyword list, the various sections in the RFC document containing the message format content, the finite state machine content, and the message format content and the finite state machine content are determined according to the keyword matching method as the various preliminary selected sections, and the corresponding relationship between each preliminary selected section and the content contained therein is established, and then step B is entered;

[0016] Step B. For each of the preliminary selected chapters, apply the target large language model to verify whether the correspondence between the preliminary selected chapters determined in step A and the contents contained therein is correct, wherein the preliminary selected chapters for which the correspondence is incorrect constitute the revised chapters and are added to the candidate set; the preliminary selected chapters for which the correspondence is correct constitute the selected chapters, and then proceed to step C;

[0017] In the above step B, for each of the preliminary selected chapters, perform the following steps B1 to B2;

[0018] Step B1. Input the preliminary selected chapters and the corresponding relationship between them and the included content into the target large language model, and send an instruction to the target large language model to determine whether the input content is correct. The target large language model analyzes the received instruction based on the received input content. If the feedback judgment is correct, the preliminary selected chapters constitute the chapters to be selected, and enter step B2; if the feedback judgment is incorrect, the preliminary selected chapters are the preliminary selected chapters whose corresponding relationship is verified to be incorrect in step B;

[0019] Step B2. Sending an instruction to the target large language model again to inquire whether its judgment on the candidate chapter in step B1 is correct, and the target large language model re-analyzes its judgment on the candidate chapter in step B1. If the feedback judgment is correct, the candidate chapter constitutes the selected chapter; if the feedback judgment is incorrect, the candidate chapter is the preliminary selected chapter whose corresponding relationship is incorrectly verified in step B;

[0020] Step C. For each selected chapter, apply the target large language model to analyze whether the selected chapter has the corresponding extraction fields of the included content according to the preset extraction fields of each type of content, wherein the selected chapters that do not have the corresponding extraction fields of the included content are analyzed to form the revised chapters and added to the candidate set; the selected chapters that have the corresponding extraction fields of the included content are analyzed to form the target chapters; then proceed to step D;

[0021] Step D. Applying the target large language model to extract the extracted fields of the content corresponding to each target chapter according to the preset extracted fields of each type of content, combining and constituting the message format and finite state machine of the network protocol in the RFC document, and then proceeding to step E;

[0022] Step E. Based on each revised chapter in the candidate set, the target large language model is applied to construct an error instance, and the target large language model performs correction processing on each extracted field corresponding to the content contained in each target chapter extracted in step D according to the error instance, and updates the message format and finite state machine that constitute the network protocol in the RFC document.

[0023] As a preferred technical solution of the present invention: in the step A, according to the preset network protocol specification keyword list, keyword matching is performed on the RFC document in a regular expression matching manner to determine the chapters in the RFC document that respectively contain message format content, finite state machine content, and message format content and finite state machine content as the preliminary chapters.

[0024] As a preferred technical solution of the present invention: the extraction fields preset for each type of content in step C include the following:

[0025] Each extracted field of the preset message format content includes each field name in the network protocol message format, each field length, and description content corresponding to each field;

[0026] Each extracted field of the preset finite state machine content includes an original state, a target state, and a condition for triggering state transfer in the network protocol finite state machine.

[0027] As a preferred technical solution of the present invention: the step E comprises the following steps;

[0028] Step E1. Randomly select two correction chapters from the candidate set and input them into the target large language model, and send an instruction to the target large language model to construct an error instance that does not correspond between the message format and the finite state machine. The target large language model executes the received instruction based on the two received correction chapters to obtain the error instance, and then proceeds to step E2;

[0029] Step E2. Sending an instruction to summarize the cause of the error for the error instance to the target large language model, the target large language model analyzes the error instance, outputs the cause of the error, and proceeds to step E3;

[0030] Step E3. Send an instruction to the target large language model to correct the extracted fields of the content corresponding to each target chapter extracted in step D according to the cause of the error. The target large language model corrects the extracted fields of the content corresponding to each target chapter, and then updates the message format and finite state machine that constitute the network protocol in the RFC document.

[0031] The method for automatically extracting network protocol specifications based on a large language model described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0032] The present invention designs a method for automatically extracting network protocol specifications based on a large language model. First, each chapter in the RFC document corresponding to the network protocol message format content and the finite state machine content is determined in a matching manner. Then, a target large language model is applied to perform secondary confirmation of the corresponding relationship between each chapter on the network protocol, and corresponding extraction fields are analyzed to determine each target chapter. Finally, the target large language model is applied to extract each extraction field in each target chapter, and the combination constitutes the message format and finite state machine of the network protocol in the RFC document. The design method overcomes the limitations of the prior art and supports the automated understanding and analysis of the network protocol to achieve more accurate and efficient network protocol extraction.

[0033] The present invention designs an automatic extraction method for network protocol specifications based on a large language model, designs an error handling mechanism, reduces the occurrence of LLMs hallucination phenomenon by asking questions again, constructs error instances to simulate inconsistencies, summarizes the root causes of the error instances, and corrects the extraction of each extraction field in each target chapter to ensure the logical consistency and accuracy between the message format and the finite state machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a structural schematic diagram of a method for automatically extracting network protocol specifications based on a large language model designed by the present invention;

[0035] Figure 2 It is a flowchart of step E in the method for automatically extracting network protocol specifications based on a large language model designed by the present invention. DETAILED DESCRIPTION

[0036] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0037] The present invention designs a method for automatically extracting network protocol specifications based on a large language model, such as Figure 1 As shown, perform the following steps to extract the message format and finite state machine of the network protocol in the RFC document.

[0038] Step A. According to the preset network protocol specification keyword list, the RFC document is matched with keywords in a regular expression matching manner to determine the chapters in the RFC document that contain the message format content (1), the finite state machine content (2), and the message format content and the finite state machine content (1, 2) as the preliminary chapters, and establish the corresponding relationship between each preliminary chapter and the content it contains, and then proceed to step B.

[0039] The preset network protocol specification keyword list here mainly includes professional terms and phrases closely related to the protocol specification. The various preliminary selected chapters determined include three categories, corresponding to chapters containing message format content (1), corresponding to chapters containing finite state machine content (2), and corresponding to chapters containing message format content and finite state machine content (1, 2). Since chapters containing multiple extraction contents at the same time are more prone to errors during the extraction process, the present invention classifies chapters with different contents as described above to improve the accuracy and efficiency of the overall extraction.

[0040] Step B. For each of the preliminary selected chapters, apply the existing trained target large language model to verify whether the correspondence between the preliminary selected chapters determined in step A and the contents contained therein is correct, wherein the preliminary selected chapters for which the verification correspondence is incorrect constitute revised chapters and are added to the candidate set; the preliminary selected chapters for which the verification correspondence is correct constitute selected chapters, and then proceed to step C.

[0041] In actual application, the above step B is performed for each preliminary selected chapter respectively, and the following steps B1 to B2 are performed.

[0042] Step B1. Input the preliminary selected chapters and the correspondence between them and the content they contain into the target large language model, and send an instruction to the target large language model to determine whether the input content is correct. The target large language model analyzes the received instruction based on the received input content. If the feedback is correct, the preliminary selected chapters constitute the chapters to be selected, and proceed to step B2; if the feedback is incorrect, the preliminary selected chapters are the preliminary selected chapters whose correspondence was verified to be incorrect in step B.

[0043] Step B2. Send an instruction to the target large language model again to inquire whether its judgment on the candidate chapter in step B1 is correct. The target large language model will re-analyze its judgment on the candidate chapter in step B1. If the feedback is correct, the candidate chapter will constitute the selected chapter. If the feedback is incorrect, the candidate chapter will be the preliminary selected chapter whose corresponding relationship was verified to be incorrect in step B.

[0044] The design of step B above introduces an error handling mechanism. By asking questions again, the occurrence of LLMs hallucination phenomenon is reduced. The error handling mechanism requires the target large language model to confirm the correspondence between chapters and their contents twice to reduce the occurrence of large model hallucination phenomenon. The confirmation here is not a simple repetition, but requires the target large language model to re-evaluate the specific questions raised. Through the error handling mechanism, it ensures consistency with the actual content in the RFC document, effectively reduces the error information generated by the model due to misunderstanding or incorrect interpretation of the document content, improves the reliability of information extraction, and ensures that the final output result matches the actual content of the RFC document.

[0045] In the actual design and implementation of step B above, the answers of the target large language model will be limited to simple "yes" or "no", which can reduce uncertainty and make it easier to automate, so as to quickly identify and correct potential errors.

[0046] Step C. For each selected chapter, apply the target large language model to analyze whether the corresponding extraction fields of the included content exist in the selected chapter according to the preset extraction fields of each type of content. Among them, the selected chapters without the corresponding extraction fields of the included content are analyzed to constitute the revised chapters and added to the candidate set; the selected chapters with the corresponding extraction fields of the included content are analyzed to constitute the target chapters, and then enter step D.

[0047] In actual applications, the extracted fields of the preset message format content include the field names, field lengths, and description contents corresponding to the fields in the network protocol message format; the extracted fields of the preset finite state machine content include the original state, target state, and conditions for triggering state transition in the network protocol finite state machine.

[0048] In the analysis process in the above step C, the large language model designed by the present invention also follows the strict format requirements to answer, and does not use complete sentences, but presents information in a concise list form. If the corresponding extracted fields do not exist in a selected chapter, the large language model designed by the present invention returns "NONE" to clearly indicate the lack of information, and constitutes each revised chapter and adds it to the candidate set.

[0049] Step D. Apply the target large language model to extract the extraction fields of the content corresponding to each target chapter according to the preset extraction fields of each type of content, combine them to form the message format and finite state machine of the network protocol in the RFC document, and then enter step E.

[0050] After the above steps, the hallucination problem of extracting RFC document information by the target large language model has been greatly reduced, but there is still a problem of mismatch between the message format and the finite state machine, that is, further design and execute the following step E.

[0051] Step E. Based on each revised chapter in the candidate set, the target large language model is applied to construct an error instance, and the target large language model performs correction processing on each extracted field corresponding to the content contained in each target chapter extracted in step D according to the error instance, and updates the message format and finite state machine that constitute the network protocol in the RFC document.

[0052] In practical applications, such as Figure 2 As shown, the above step E is specifically designed to execute the following steps E1 to E3.

[0053] Step E1. Randomly select two corrected chapters from the candidate set and input them into the target large language model, and send an instruction to the target large language model to construct an error instance that does not correspond between the message format and the finite state machine. The target large language model executes the received instruction based on the two corrected chapters received, obtains the error instance, and then enters step E2.

[0054] Step E2. Send an instruction to the target large language model to summarize the cause of the error for the error instance. The target large language model analyzes the error instance, outputs the cause of the error, and proceeds to step E3.

[0055] Step E3. Send an instruction to the target large language model to correct the extracted fields of the content corresponding to each target chapter extracted in step D according to the cause of the error. The target large language model corrects the extracted fields of the content corresponding to each target chapter, and then updates the message format and finite state machine that constitute the network protocol in the RFC document.

[0056] The above design ensures a one-to-one correspondence between the extracted message format and the finite state machine information by encouraging the target large language model to think about known errors and thus improve the previous output results.

[0057] In the above step E, an error instance is constructed to simulate the inconsistency. Based on summarizing the root causes of the error instance, the extraction of each extraction field in each target chapter is corrected to ensure the logical consistency and accuracy between the message format and the finite state machine. The analysis process of the error instance involves the review of the document content and the reflection on the model extraction logic. By identifying the specific factors that cause the inconsistency, the target large language model summarizes the cause of the error and proposes targeted strategies to improve the information extraction process, aiming to optimize the model's extraction logic and reduce or eliminate the inconsistency between the message format and the finite state machine.

[0058] The automatic extraction method of network protocol specifications based on a large language model designed by the present invention first determines the various chapters in the RFC document corresponding to the network protocol message format content and the finite state machine content in a matching manner, then applies the target large language model to perform secondary confirmation on the correspondence between the various chapters regarding the network protocol, and analyzes the corresponding extraction fields to determine the various target chapters, and finally applies the target large language model to extract the various extraction fields in the various target chapters, which are combined to form the message format and finite state machine of the network protocol in the RFC document. The design method overcomes the limitations of the prior art and supports the automated understanding and analysis of the network protocol to achieve more accurate and efficient network protocol extraction.

[0059] The design method of the present invention uses a large language model in the network protocol extraction stage, which significantly improves the ability to accurately extract message formats and finite state machines from RFC documents, greatly reduces dependence on manual operations, reduces errors caused by personal understanding deviations, and at the same time improves the extraction efficiency and the quality and reliability of the results, providing strong support for the automated understanding and analysis of network protocols, and becoming the key to achieving efficient and accurate information extraction.

[0060] In the stage of extracting RFC document information, the hallucination problem of large language models is effectively reduced, that is, the risk of content not mentioned in the model-generated document, ensuring the accuracy and reliability of the extracted information, providing a solid foundation for the automated understanding and analysis of network protocols, and reflecting the foresight and innovation of the present invention in improving the quality of automated extraction.

[0061] In the RFC document extraction process, a consistency alignment strategy is used, which ensures the logical consistency between the message format extracted from the RFC document and the finite state machine, identifies and summarizes the inconsistencies between the two, guides model learning and self-correction, and thus improves the accuracy and reliability of information extraction. This stage not only optimizes model performance and reduces subsequent correction costs, but also enhances the generalization ability of the model, improves users' trust in automated extraction tools, and supports the processing of complex documents, significantly improving the intelligence level of the automated extraction process and reducing the need for manual intervention.

[0062] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for automatically extracting network protocol specifications based on a large language model, characterized by: Perform the following steps to extract the message format and finite state machine of the network protocol in the RFC document; Step A. According to the preset network protocol specification keyword list, the various sections in the RFC document containing the message format content, the finite state machine content, and the message format content and the finite state machine content are determined according to the keyword matching method as the various preliminary selected sections, and the corresponding relationship between each preliminary selected section and the content contained therein is established, and then step B is entered; Step B. For each of the preliminary selected chapters, apply the target large language model to verify whether the correspondence between the preliminary selected chapters determined in step A and the contents contained therein is correct, wherein the preliminary selected chapters for which the correspondence is incorrect constitute the revised chapters and are added to the candidate set; the preliminary selected chapters for which the correspondence is correct constitute the selected chapters, and then proceed to step C; In the above step B, for each of the preliminary selected chapters, perform the following steps B1 to B2; Step B1. Input the preliminary selected chapters and the corresponding relationship between them and the included content into the target large language model, and send an instruction to the target large language model to determine whether the input content is correct. The target large language model analyzes the received instruction based on the received input content. If the feedback judgment is correct, the preliminary selected chapters constitute the chapters to be selected, and enter step B2; if the feedback judgment is incorrect, the preliminary selected chapters are the preliminary selected chapters whose corresponding relationship is verified to be incorrect in step B; Step B2. Sending an instruction to the target large language model again to inquire whether its judgment on the candidate chapter in step B1 is correct, and the target large language model re-analyzes its judgment on the candidate chapter in step B1. If the feedback judgment is correct, the candidate chapter constitutes the selected chapter; if the feedback judgment is incorrect, the candidate chapter is the preliminary selected chapter whose corresponding relationship is incorrectly verified in step B; Step C. For each selected chapter, apply the target large language model to analyze whether the selected chapter has the corresponding extraction fields of the included content according to the preset extraction fields of each type of content, wherein the selected chapters that do not have the corresponding extraction fields of the included content are analyzed to form the revised chapters and added to the candidate set; the selected chapters that have the corresponding extraction fields of the included content are analyzed to form the target chapters; then proceed to step D; Step D. Applying the target large language model to extract the extracted fields of the content corresponding to each target chapter according to the preset extracted fields of each type of content, combining and constituting the message format and finite state machine of the network protocol in the RFC document, and then proceeding to step E; Step E. Based on each revised chapter in the candidate set, the target large language model is applied to construct an error instance, and the target large language model performs correction processing on each extracted field corresponding to the content contained in each target chapter extracted in step D according to the error instance, and updates the message format and finite state machine that constitute the network protocol in the RFC document.

2. According to claim 1, a method for automatically extracting network protocol specifications based on a large language model is characterized by: In step A, keyword matching is performed on the RFC document in a regular expression matching manner according to a preset network protocol specification keyword list to determine the chapters in the RFC document that respectively contain message format content, finite state machine content, and message format content and finite state machine content as preliminary chapters.

3. The method for automatically extracting network protocol specifications based on a large language model according to claim 1, characterized in that: The extraction fields of various types of content preset in step C include the following: Each extracted field of the preset message format content includes each field name in the network protocol message format, each field length, and description content corresponding to each field; Each extracted field of the preset finite state machine content includes an original state, a target state, and a condition for triggering state transfer in the network protocol finite state machine.

4. The method for automatically extracting network protocol specifications based on a large language model according to claim 1, characterized in that: The step E comprises the following steps: Step E1. Randomly select two correction chapters from the candidate set and input them into the target large language model, and send an instruction to the target large language model to construct an error instance that does not correspond between the message format and the finite state machine. The target large language model executes the received instruction based on the two received correction chapters to obtain the error instance, and then proceeds to step E2; Step E2. Sending an instruction to summarize the cause of the error for the error instance to the target large language model, the target large language model analyzes the error instance, outputs the cause of the error, and proceeds to step E3; Step E3. Send an instruction to the target large language model to correct the extracted fields of the content corresponding to each target chapter extracted in step D according to the cause of the error. The target large language model corrects the extracted fields of the content corresponding to each target chapter, and then updates the message format and finite state machine that constitute the network protocol in the RFC document.

Images