Intelligent protocol auditing method, device and equipment, and storage medium

Abstract

The application discloses an intelligent protocol auditing method and device, equipment and a storage medium, and relates to the technical field of data auditing. The method comprises the following steps: obtaining a to-be-audited protocol corresponding to a current auditing task; based on target task information corresponding to the to-be-audited protocol, screening a guide example from a preset example set; inputting the to-be-audited protocol and the guide example into a preset protocol auditing model to obtain an initial auditing result; obtaining feedback information of a user on the initial auditing result, and updating the preset example set according to the feedback information, so that the preset protocol auditing model generates a final auditing result based on the updated preset example set.

Description

Technical Field

[0001] This application relates to the field of data auditing technology, and in particular to a smart protocol auditing method, apparatus, device and storage medium. Background Technology

[0002] Intelligent protocol review technology based on large models can adopt a static Few-Shot architecture. Few-Shot refers to the technique of guiding the model to learn the inherent rules of the task through a small number of samples. In existing intelligent protocol review technologies, static Few-Shot can be used to add output samples to the prompts of the large model to improve the review quality.

[0003] However, existing smart protocol review solutions have fixed example libraries, which cannot adapt to the complex content of the review protocols. This can easily lead to situations where large models simply repeat the example content, resulting in a high rate of illusion in the protocol review results output by the large models and insufficient reliability of the output.

[0004] Therefore, how to improve the reliability of protocol auditing schemes based on large models has become an urgent problem to be solved. Summary of the Invention

[0005] The main objective of this application is to provide an intelligent protocol auditing method, apparatus, device, and storage medium, aiming to solve the technical problem of how to improve the reliability of protocol auditing schemes based on large models.

[0006] To achieve the above objectives, this application proposes a smart protocol auditing method, which includes: Retrieve the pending agreement corresponding to the current collaborative review task; Based on the target task information corresponding to the protocol to be reviewed, guide examples are selected from a preset example set; Input the protocol to be reviewed and the guidance example into the preset protocol review model to obtain the initial review result; Obtain user feedback on the initial review result, and update the preset example set based on the feedback, so that the preset protocol review model generates the final review result based on the updated preset example set.

[0007] Furthermore, to achieve the above objectives, this application also proposes an intelligent protocol review device, which includes: The document acquisition module is used to acquire the agreement to be reviewed corresponding to the current collaborative review task; The example filtering module is used to filter guiding examples from a preset example set based on the target task information corresponding to the protocol to be reviewed; The preliminary review module is used to input the protocol to be reviewed and the guidance example into a preset protocol review model to obtain the initial review result; The example optimization module is used to obtain user feedback information on the initial review result and update the preset example set according to the feedback information, so that the preset protocol review model generates the final review result based on the updated preset example set.

[0008] In addition, to achieve the above objectives, this application also proposes an intelligent protocol auditing device, which includes: a memory, a processor, and an intelligent protocol auditing program stored in the memory and executable on the processor. The intelligent protocol auditing program is configured to implement the steps of the intelligent protocol auditing method described above.

[0009] In addition, to achieve the above objectives, this application also provides a storage medium storing a program that implements the smart protocol auditing method, the program implementing the smart protocol auditing method being executed by a processor to implement the steps of the smart protocol auditing method as described above.

[0010] This application provides a smart protocol review method, apparatus, device, and storage medium. The method includes: obtaining the protocol to be reviewed corresponding to the current collaborative review task; selecting guiding examples from a preset example set based on the target task information corresponding to the protocol to be reviewed; inputting the protocol to be reviewed and the guiding examples into a preset protocol review model to obtain an initial review result; obtaining user feedback information on the initial review result, and updating the preset example set according to the feedback information, so that the preset protocol review model generates a final review result based on the updated preset example set.

[0011] This application first dynamically filters associated guiding examples from a preset example set based on the target task information corresponding to the protocol to be reviewed, and then updates the preset example library of the large model in a closed loop based on user feedback on the initial review results, thereby improving the dynamic adaptability of the preset example set of the large model to the collaborative review task, and thus improving the reliability of the protocol review results generated by the large model iteratively based on the updated preset example set. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating the first embodiment of the smart protocol review method of this application; Figure 2 This is a schematic diagram of the information preprocessing process in the first embodiment of the smart protocol review method of this application; Figure 3 This is a schematic diagram illustrating the adaptive optimization process of the smart protocol review method in this application. Figure 4 This is a flowchart illustrating the second embodiment of the smart protocol review method of this application; Figure 5 This is a flowchart illustrating the optimized process of the smart protocol review method in this application; Figure 6 A comparative diagram illustrating the optimization of model invocation in the smart protocol review method of this application; Figure 7 This is a schematic diagram of the module structure of the intelligent protocol review device according to an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the smart protocol auditing method in this application embodiment.

[0015] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0016] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application. Furthermore, all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the corresponding device.

[0017] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0018] The main solution of this application is: to obtain the protocol to be reviewed corresponding to the current collaborative review task; to select guiding examples from the preset example set based on the target task information corresponding to the protocol to be reviewed; to input the protocol to be reviewed and the guiding examples into the preset protocol review model to obtain the initial review result; to obtain the user's feedback information on the initial review result, and to update the preset example set according to the feedback information.

[0019] Currently, existing intelligent protocol review technologies suffer from two problems. First, Few-Shot samples are not well-suited and lack freshness. Due to the token (tag) length limitation of the large model, only a small number of output example samples are suitable for insertion into the Prompt. When the input protocol does not match the sample content, the review accuracy of the large model will decrease. Moreover, the "required sensitive user information" in the sample content is a frequently changing business maintenance field, while the quality of Few-Shot examples relies on manual annotation and cannot be dynamically optimized based on actual usage effects. Therefore, when the business of the review protocol is updated, the suitability and freshness of Few-Shot samples will be further reduced. Second, inappropriate samples can easily induce the illusion of a Large Language Model (LLM). The content involved in collaborative review tasks is quite complex. Although existing general-purpose LLMs can rely on the examples in the Prompt to understand the task objectives and internal processing logic, when the examples differ significantly from the user input, the LLM may exhibit the illusion of mixed Prompt examples in the answer or repeated Prompt examples. Therefore, existing intelligent protocol review technologies based on large models suffer from the problem of a fixed static Few-Shot example library, which cannot adapt to changes in user input (such as updates to sensitive information fields). This leads to the large model being prone to mechanically rereading example content, resulting in a high rate of illusion in the output results and insufficient output reliability.

[0020] To address this issue, this application first dynamically filters relevant guiding examples from a pre-set example set based on the target task information corresponding to the protocol to be reviewed, and then updates the pre-set example library of the large model in a closed loop based on user feedback on the initial review results. Therefore, this application can dynamically filter samples based on association information and adjust the credibility of the large model example samples in real time based on user feedback, ensuring that the input samples are always relevant to the current task and of high quality, thereby significantly reducing model illusion and improving the accuracy of collaborative review.

[0021] It should be noted that the executing entity in this embodiment can be an intelligent protocol review system, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an intelligent protocol review device capable of performing the above functions. This embodiment does not specifically limit this. The following uses an intelligent protocol review device (hereinafter referred to as the review device) as the executing entity to describe this embodiment and the following embodiments.

[0022] Based on this, this application provides an intelligent protocol review method. This method is an automated intelligent protocol review method for text compliance scenarios such as mini-program access and authorization agreements. It is based on a large language model and combines dynamic example filtering, model generation, and feedback self-iteration. It can be applied to institutions such as banks and internet platforms to conduct automated compliance reviews of the consistency of sensitive information in user authorization agreements, privacy agreements, and terms of service. The process focuses on comparing the consistency between the authorized sensitive information in the agreement and the sensitive information registered by the user.

[0023] At this time, refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the smart protocol review method of this application.

[0024] In this embodiment, the smart protocol auditing method includes steps S10 to S40: Step S10: Obtain the agreement to be reviewed corresponding to the current collaborative review task; It is understood that the aforementioned current collaborative review task can refer to a single agreement compliance review business unit initiated by a user through a mini-program or platform, etc. This current collaborative review task may include task attributes such as the agreement to be reviewed, business scenario identifier, and required sensitive authorization information. In this case, the aforementioned agreement to be reviewed can refer to the agreement content that needs to be reviewed for compliance in the current collaborative review task, and may include all terms of the agreement and sensitive information authorization content. Furthermore, in this embodiment, the agreement to be reviewed can be in plain text form or in the form of a web link.

[0025] It is easy to understand that existing intelligent review technology only supports plain text protocol input. Protocols submitted for review in the form of web links typically have diverse data types, including not only text but also potentially images. Therefore, the existing intelligent review process cannot currently handle protocols in the form of links. However, in actual business operations, a large number of users submit protocols in the form of links, resulting in these protocols requiring manual review and reducing review efficiency. In this case, in a feasible implementation, in this embodiment, step S10 may include steps S11~S13: Step S11: Receive the input document information corresponding to the current collaborative review task, and determine whether the input document information is in the form of a link; Understandably, the aforementioned input document information refers to the original protocol content submitted by the user for the current collaborative review task. This can be a plain text string or a protocol link such as HTTP / HTTPS. In this case, the review device can check if the user's input content begins with http: / / or https: / / . If so, it can be determined to be protocol information in the form of a link, and a link conversion process can be executed; otherwise, it can be determined to be protocol information in plain text or other formats that do not require further processing, and the input document information can be directly used as the protocol to be reviewed.

[0026] Step S12: If yes, crawl the page source code corresponding to the input document information, perform text recognition and conversion on the non-text content in the page source code, and concatenate the converted text information with the text content of the page source code to generate an agreement to be reviewed. Step S13: If not, then directly use the protocol content corresponding to the input document information as the protocol to be reviewed.

[0027] It is easy to understand that the above page source code can refer to the complete HTML (Hyper Text Markup Language) source code corresponding to the protocol link obtained by simulating a browser accessing the link through web crawling technology. It may contain the text, images, styles and scripts of the web page.

[0028] At this point, the aforementioned non-text content can refer to images, videos, audio, and other content in the page source code that cannot be directly processed by general LLM. In this embodiment, the non-text content mainly refers to images containing protocol text. Therefore, the aforementioned text recognition and conversion can refer to the operation of using OCR (Optical Character Recognition) technology to extract the text from the image and convert it into editable text content.

[0029] It should be noted that in the above splicing process, the review device can first obtain the protocol link containing each image and its corresponding image tag, and then use an OCR tool (such as Tesseract, PaddleOCR, or a cloud-based OCR service) to recognize the text in each image. Then, the review device can splice the image text recognized by the OCR tool to the corresponding image tag in the original protocol link to obtain the complete protocol text, i.e., the protocol to be reviewed.

[0030] For example, the review device can first use BeautifulSoup to parse the HTML source code and find all images. Tags; then iterate through each Tags are used to download the corresponding image files; then PaddleOCR is used to perform text recognition on the images to obtain the text content; finally, each element in the HTML source code is extracted. The tags are replaced with the corresponding OCR-recognized text; finally, all plain text content in the HTML is extracted, and extra spaces, line breaks, and HTML tags are removed, and the text is concatenated into a complete agreement text to be reviewed.

[0031] Furthermore, if the review device detects image download failure or OCR recognition failure, it can retry twice. If the problem persists after all retries have been exhausted, it can switch to a multimodal large model (such as GPT-4V) for data augmentation and image text extraction. If the multimodal large model also fails, the original image can be retained in the pending review protocol to be stitched together. The protocol text is tagged and an error note is added at the beginning: "[Note] Some image content cannot be recognized and requires manual review." This embodiment proposes a multiple error handling mechanism to ensure that even if image recognition fails, an auditable protocol text can still be generated, prompting for manual review.

[0032] Therefore, this embodiment supports both text and link input formats, enabling broader coverage of intelligent review scenarios; moreover, this embodiment can automatically complete link crawling, image OCR, and text splicing without manual intervention, further saving labor costs.

[0033] Furthermore, existing large models cannot effectively handle excessively long protocols (such as privacy statements and terms of service) exceeding the LLM context length, making it impossible to automatically review these protocols and requiring manual processing. In this case, in a feasible implementation, steps A1-A3 may be included before step S20: Step A1: If the amount of data in the protocol to be reviewed exceeds the processing threshold, the protocol to be reviewed is divided into multiple document fragments in logical order. Understandably, the aforementioned processing threshold may refer to the maximum text length (e.g., 4096 characters) that the subsequent preset protocol review model can effectively process. For protocols exceeding this length, the review device needs to perform segmentation and summarization processing in advance to ensure the processing accuracy of the model.

[0034] Therefore, the aforementioned document fragments can refer to multiple short text fragments obtained by dividing the long protocol in a logical order, and the length of each fragment must be controlled within the processing threshold.

[0035] For example, if the review device detects that the data volume of the protocol to be reviewed does not exceed 4096 characters, it can directly proceed to the subsequent example filtering, matching and review process; if the review device detects that the data volume of the protocol to be reviewed exceeds 4096 characters, it can be divided into paragraphs of the protocol to be reviewed, and the length of each document segment can be controlled between 1000-2000 characters; when dividing, an additional 50 characters can be reserved above and below each of the two adjacent segments to prevent sentences from being truncated and causing information to be missed.

[0036] Step A2: Extract key information from each document fragment input to obtain summary information corresponding to each document fragment; Step A3: Assemble the summary information according to the logical order to obtain the optimized information document, and use the optimized information document as the new agreement to be reviewed.

[0037] Understandably, the aforementioned key information extraction can refer to the operation of extracting key information related to agreement review from each segmented document fragment, such as sensitive information authorization clauses, liability for breach of contract, and validity period. In this case, the aforementioned summary information can be a condensed text of the key information corresponding to each document fragment, retaining all the core content required for review in each document fragment.

[0038] It's important to understand that the aforementioned optimized information document can refer to a simplified version of the protocol text obtained by assembling the summary information of all fragments in the logical order of the original protocol. In this case, the review device can use the optimized information document as a new protocol to be reviewed and input it into the subsequent example filtering, matching, and review process.

[0039] Therefore, this embodiment can ensure that the preset protocol model can automatically process protocols of any length through segmentation and digestion techniques, reducing manual costs; and the length of the simplified information optimization document is greatly shortened, which can effectively reduce model inference time and token consumption, and improve the review speed.

[0040] At this point, for ease of understanding, based on Figure 2 The information preprocessing process of this embodiment will be explained and described. Figure 2 This is a schematic diagram of the information preprocessing process in the first embodiment of the smart protocol review method of this application. Figure 2 As shown, after receiving the uploaded protocol (i.e., the protocol to be reviewed), for protocols in the form of web links, since some pages display the protocol using images, the review device can obtain the valid protocol text to be reviewed based on crawling and OCR text parsing. At the same time, the review device can also perform segmented summarization for excessively long protocols to adapt to the LLM input length limit. Finally, the review device can input the preprocessed text into the preset protocol review model (i.e., the co-review Agent in the figure) to perform sensitive word review, authorization information extraction and comparison in sequence, and generate the final review result.

[0041] At this time, as Figure 2 As shown, the review device can first determine whether the uploaded protocol is in the form of a webpage link. If it is, in the link conversion stage, it can first crawl the HTML source code, that is, use a crawler tool to simulate a browser to access the link, obtain the complete HTML source code of the webpage, and restore the original content of the page; then perform image OCR conversion, perform text recognition on the image containing the protocol text in the HTML, and extract the text content in the image; then perform protocol text concatenation, concatenate the text recognized by OCR with the text content of the webpage, remove HTML tags and irrelevant formatting, and generate a complete plain text protocol; If the protocol is not in the form of a link, the auditing device can proceed directly to the next step, checking the length of the protocol text to determine if it exceeds the maximum token limit that the preset protocol auditing model can handle. If so, the data cleaning stage begins. First, the protocol text is segmented according to logical paragraphs, with each segment's length controlled within the model's processing range, while preserving overlapping content between segments to prevent key information from being truncated. Next, a key information summary is generated, extracting core content related to sensitive information authorization (such as authorization information type, purpose, scope, and duration) from each text segment to generate a concise summary. Finally, the protocol summary is assembled, concatenating the summaries of each segment according to the original protocol's logical order to form a concise protocol text with complete information and controllable length. In this case, this embodiment can adapt to the context length limit of LLM for ultra-long protocols such as privacy statements through segmentation + summarization technology without losing key information.

[0042] If the length of the protocol text does not exceed the processing threshold, it can directly proceed to the subsequent collaborative review Agent stage.

[0043] Step S20: Based on the target task information corresponding to the protocol to be reviewed, select guiding examples from the preset example set; Understandably, the aforementioned preset example set can refer to a pre-stored example library used to guide the large model to output correct review results. Furthermore, each example in the preset example set can contain a complete triple of "protocol content - required sensitive information - review result," used to store the protocol content, required sensitive information, and standard review result corresponding to each example. Each example is also associated with a credibility parameter. In this embodiment, the credibility parameter of each example in the preset example set can be automatically iteratively updated based on user feedback.

[0044] The aforementioned target task information can characterize the core features of this review, and may include the set of sensitive authorization information fields required for the current collaborative review task. The review device can obtain the agreement to be reviewed and the sensitive information (target task information) selected by the user from the review application submitted by the user.

[0045] Furthermore, this embodiment can filter examples containing the same or similar sensitive information related to the current collaborative review task from a preset example set maintained by the large model based on the target task information. In this case, the aforementioned guiding example can refer to the example selected from the preset example set and embedded in the large model's prompt word to guide the large model's output. This preset example can represent several examples most relevant to the current collaborative review task and with the highest credibility, typically the top K credibility examples in the preset example set, where K can be 1 to 5.

[0046] Step S30: Input the protocol to be reviewed and the guidance example into the preset protocol review model to obtain the initial review result; Understandably, the aforementioned pre-defined protocol review model can refer to a large language model or rule engine that acts as a "Generator" during the review process, responsible for generating the protocol review results corresponding to the protocol to be reviewed, and which has been pre-tuned or optimized through a Prompt process. In this case, the aforementioned initial review result can refer to the protocol review result output by the pre-defined protocol review model for the first time, including sensitive information comparison results, consistency conclusions, and processing suggestions, usually in JSON format.

[0047] Therefore, in this embodiment, the auditing device can construct a structured prompt containing system prompts, guidance examples, and the protocol to be audited. Then, based on the structured prompts, it calls a preset protocol auditing model to generate an initial audit result in JSON format. For example, the structured prompts can be represented as: system prompts (requiring comparison of the protocol with the registered sensitive information, outputting a specified JSON format) + 3 guidance examples + protocol to be audited + target task information.

[0048] At this time, as Figure 2 As shown, the core review process executed by the preset agreement review model may include: First, performing sensitive word review by scanning the sensitive information keywords (such as name, mobile phone number, ID card number, biometric information, etc.) in the agreement text to be reviewed, and locating all clauses involving user information authorization; then, extracting authorization information by extracting key elements of the authorization information from the agreement, including the type of authorized information, purpose of use, scope of use, storage period, and third-party sharing status; finally, comparing the authorization information by comparing the extracted agreement authorization information with the sensitive information required by the user registered on the platform item by item to determine whether it exceeds the scope of authorization, whether there are any illegal collection or use clauses, and generating an initial review result. Figure 2 The results of the joint review.

[0049] Understandably, due to the concurrency limitations of large models in actual operation, the JSON strings output by existing intelligent protocol review schemes based on large models contain occasional internal model errors such as missing characters, which can easily lead to review failures. Therefore, the usability of intelligent protocol review needs further improvement. In a feasible implementation, this embodiment may further include steps B1-B4 after step S30: Step B1: The initial review results are scored using a verification model to obtain a collaborative review result score and optimization suggestions. Understandably, the aforementioned verification model can be another LLM (or a combination of a rule engine and an LLM) in the auditing device that plays the role of "Reflector" during the auditing process. It can be responsible for quality checks and scoring of the preliminary or new audit results generated by Generator (i.e., the aforementioned preset protocol auditing model), generating corresponding collaborative audit result scores and optimization suggestions. That is, in this embodiment, when the preset protocol auditing model has output errors, resource unavailability, or low output quality, the verification model can detect these issues and, with optimization suggestions, enable the preset protocol auditing model to regenerate iterative audit results.

[0050] For example, after the auditing device inputs the initial audit results, the agreement to be audited, and the required sensitive information into the verification model, the verification model can check the following from both data structure and semantic dimensions during the quality scoring process: 1. Check the data structure for correctness. The validation model can first check whether the output data structure of the initial review results is correct through methods such as JSON conversion and regular expression matching. If the format is incorrect, the score is 0 (because the collaborative review results are unusable if the data structure is incorrect), and the optimization suggestion can be "JSON format error, please correct the parentheses, quotation marks or commas"; if the format is correct, proceed to the next step of semantic check; 2. Check the semantic correctness of the data. The validation model can further check whether all the required sensitive information has been compared, whether the comparison results are consistent with the protocol content, and whether the processing suggestions are reasonable, giving a score of 0-1 and specific optimization suggestions.

[0051] At this point, the aforementioned collaborative review result score can refer to the comprehensive score given by the verification model to the review result, with a value range of [0, 1]. A higher score indicates better result quality. The aforementioned optimization suggestions can refer to the problems and improvement directions identified by the verification model in the review results, such as JSON format errors, omissions of sensitive information, and incorrect comparison conclusions.

[0052] Step B2: Determine whether the collaborative review result score meets the quality threshold; Step B3: If the collaborative review result score does not meet the quality threshold, the optimization suggestions, the agreement to be reviewed, and the guiding example are re-input into the preset agreement review model to obtain the updated review result. Step B4: Return the updated review result as the new review result to the verification model for quality scoring again until the collaborative review result score corresponding to the new review result meets the quality threshold, and then output the new review result to the user.

[0053] Understandably, the aforementioned quality threshold can refer to a pre-set passing score for the audit result (typically set to 0.8). In this case, the audit device can only directly output the initial audit result to the user if the quality score reaches or exceeds the quality threshold, i.e., the collaborative audit result score meets the quality threshold.

[0054] If the collaborative review result score does not exceed the quality threshold, the review device can enter iterative optimization. The updated review result can refer to the review result regenerated by the preset protocol review model based on a new Prompt containing optimization suggestions. The new Prompt can be represented as: system prompt + guidance example + protocol to be reviewed + required sensitive information + optimization suggestions. At this time, the review device can repeat the verification and generation process, that is, return the new review result to the verification model for another quality score. This continues until a satisfactory result is detected (or the maximum number of iterations is reached), at which point the updated review result is output to the user. If the collaborative review result score still does not meet the standard after reaching the maximum number of iterations, the review device can select the highest score among all iteration results as the final review result output, but can also add a note to this review result stating "This result requires manual review."

[0055] For ease of understanding, please refer to Figure 3 Provide an explanation. Figure 3 This diagram illustrates the adaptive optimization process of the smart protocol review method in this application. Figure 3 As shown, this embodiment proposes an intelligent protocol review data processing flow based on a dual-model architecture of "Generator (preset protocol review model) + Reflector (verification model)". This flow can combine dynamic Few-Shot enhancement and iterative reflection optimization mechanism to realize the automatic generation, verification, optimization and selection of protocol review results.

[0056] like Figure 3 As shown, as the starting point of the process, the user authorization agreement, privacy agreement, etc., can be requested through the mini-program to trigger a collaborative review task, i.e., the current collaborative review task, which prepares for subsequent dynamic Few-Shot enhancements, sample instances, and background knowledge. At this time, the review device can extract the agreement text from the application as the original business document for this review. Figure 3 The content awaiting review is displayed in the form of web links. At this point, the review device supports automatically crawling, OCR, and text concatenation of protocols in the form of web links, and performs segmentation and summary processing on excessively long protocols to form standardized, inputtable content awaiting review.

[0057] Then, during the dynamic Few-Shot enhancement phase, the auditing device can select high-matching, high-credibility historical audit examples from a preset example set based on the target task information (business scenario or required sensitive information fields) of the current collaborative auditing task, forming a dedicated guiding example for this inference (i.e., Figure 3 The example shown is a Few-Shot sample instance, which provides a scene matching demonstration for the subsequent inference of the pre-defined protocol review model, suppresses the illusion of large models, and standardizes the output format of large models.

[0058] At this point, the specific example set after the equipment review process can include historical protocol text, corresponding sensitive information configuration, and standard review conclusions. The guiding example is then input into the preset protocol review model as part of the Prompt. Figure 3 The Generator module is shown. Simultaneously, the auditing device can combine protocol auditing-related business rules, compliance standards, field definitions, etc., into a static knowledge base (corresponding to...). Figure 3 (Based on Generator / Reflector background knowledge), it can provide underlying knowledge support for the two major models, ensuring that the review logic complies with business specifications.

[0059] In the closed-loop process of "generation-evaluation-feedback-regeneration" proposed in this embodiment, which can continuously improve the quality of collaborative review results, the Generator module can receive preprocessed content to be reviewed, Few-Shot sample instances, and background knowledge, and generate initial review results through large model inference, i.e. Figure 3 The collaborative review results shown are as follows. At this time, the output collaborative review results can include two parts: 1) Consistency comparison results, which are the judgment results of comparing the sensitive information authorized in the agreement with the required sensitive information registered on the platform item by item; 2) Collaborative review opinions, that is, the overall compliance conclusion and corresponding rectification suggestions.

[0060] Accordingly, the Reflector module (i.e. the verification model mentioned above) can receive the collaborative review results generated by the Generator, perform result evaluation, and output two types of key information: 1) optimization suggestions, which can point out the format, semantic, and logical problems in the current collaborative review results and directions for improvement; 2) collaborative review result score, which can correspond to the quantitative score representing the completeness, accuracy, and compliance of the current collaborative review results.

[0061] Next, as Figure 3 As shown, the auditing device can further determine whether the collaborative audit result score given by the verification model exceeds the threshold, i.e., the aforementioned quality threshold. If so, the auditing device can directly enter the "Collaborative Audit Quality Meets Standard" branch and directly output the collaborative audit result to the user; If not, the review device can enter the iteration process. Simultaneously, to prevent infinite loops, the review device can pre-set a maximum number of iterations (e.g., 3 times), allowing it to continue checking if the iteration limit has been reached. If the limit hasn't been reached, the review device can feed back the optimization suggestions from the Reflector to the Generator, allowing the Generator to regenerate a new review result based on the suggestions and re-enter the iteration loop. If the iteration limit has been reached, the review device can terminate the iteration and enter the collaborative review result selection branch. At this point, the review device can automatically select the highest Reflector score from all collaborative review results generated in previous iterations as the final result output to the user, ensuring that even if the target isn't met, the current optimal version is still output.

[0062] Regardless of the score, the final structured audit conclusion output to the user can include complete consistency comparison results, collaborative audit opinions, and rectification suggestions.

[0063] Therefore, this embodiment can utilize a dual-model architecture of "Generator + Reflector" to achieve self-reflection and iterative optimization, thereby automatically handling occasional format errors, semantic errors and resource failures in LLM, solving the problem of unstable and error-prone single LLM output, and significantly improving the quality and stability of audit results.

[0064] Step S40: Obtain user feedback information on the initial review result, and update the preset example set according to the feedback information, so that the preset protocol review model generates the final review result based on the updated preset example set.

[0065] It is understood that the aforementioned feedback information can refer to the evaluation results of the initial review by users such as reviewers, and may include three categories of subjective evaluation information: positive evaluation such as "accepted", negative evaluation such as "not accepted", or no evaluation. In this embodiment, this feedback information can be used to drive the update of the preset example set in a closed loop.

[0066] Therefore, the pre-set protocol review model can call the iteratively updated pre-set example set in the next round and subsequent collaborative review tasks to re-match and filter guiding examples, generating a final review result that is more business-oriented, less illusory, and more stable. At this point, the final review result has practical reference value.

[0067] Furthermore, the existing smart protocol review scheme uses a static example library, which cannot automatically add new high-quality examples or remove outdated or low-quality examples, leading to the gradual aging of the example library and a decrease in its adaptability. To solve this problem, the review device can update the credibility parameter of the guiding example used in the current sample set based on the reviewer's feedback information, and can consider the current "protocol to be reviewed - required sensitive information - initial review result" as a new example when the reviewer inputs positive evaluations such as "acceptance", and add it to the preset example set. Therefore, in a feasible implementation, this embodiment may include steps S50 to S60 after step S40: Step S50: When the feedback information meets the preset supplementary conditions, the protocol to be reviewed and the initial review result are combined into a new example, and the new example is added to the preset example set; Understandably, the aforementioned preset supplementary conditions can represent the triggering conditions for adding the review results output by the preset protocol review model as new examples to the set. In this embodiment, the preset supplementary conditions can correspond to the positive feedback information output by the reviewer. That is, the review device can check the user's feedback information. If the feedback information is detected as a positive evaluation such as "accepted", it can be determined that the preset supplementary conditions are met, and the review device can perform example supplementation operations; otherwise, if the feedback information is a negative evaluation or no evaluation, no other operations are performed on the review results.

[0068] The aforementioned new example can refer to a new example formed by standardizing the information obtained during the review process according to the fixed structure of "Agreement to be reviewed - Required sensitive information - Final review result", and saving the corresponding timestamp and initial credibility, with the initial credibility set to 0.

[0069] Furthermore, the review device can also search the preset example set for examples that are highly similar to the newly added example (similarity ≥ 90%). If such examples exist, the old example is deleted, and the credibility of the new example is initialized to 0; if not, the new example is added directly, and its credibility is initialized to 0. In this way, this embodiment can prevent the same or similar examples from being stored repeatedly through duplicate checks, saving storage space and improving screening efficiency.

[0070] Step S60: If the total number of examples corresponding to the preset example set is detected to have reached the capacity limit, a preset example elimination operation is performed based on the real-time credibility parameter of each example in the preset example set.

[0071] It is easy to understand that the aforementioned capacity limit can refer to the maximum number of examples allowed to be stored in the preset example set (e.g., 1000). This embodiment can control the memory usage and Prompt length of the auditing device based on this capacity limit. In this case, the aforementioned preset example elimination operation can refer to the operation whereby the auditing device automatically deletes the example with the lowest credibility (or, if the scores are the same, the oldest example) in the preset example set when the set capacity reaches the limit, in order to make room for subsequent new examples. That is, the auditing device can first traverse the preset example set to find the example with the lowest credibility; if multiple examples are detected to have the same and lowest credibility, their addition timestamps are compared, and the one with the earliest addition time is selected for deletion.

[0072] Therefore, this embodiment can automatically transform high-quality results adopted by reviewers into new examples, continuously enriching the coverage of the example library, thereby adapting the preset example set to new protocol types and sensitive information fields. Simultaneously, this embodiment can ensure that the example library does not expand indefinitely based on capacity limits and preset example eviction operations, while automatically removing the least trustworthy examples to maintain the overall quality of the example library.

[0073] In summary, this embodiment can dynamically filter high-confidence examples related to the current task, enabling the preset example set to quickly adapt to updates to sensitive information fields and new protocol types. This not only reduces the illusion rate of audit results output by large models and improves audit accuracy, but also eliminates the need for manual re-annotation of examples, thus reducing maintenance costs.

[0074] Furthermore, this embodiment can automatically update the credibility of examples and supplement new examples based on user feedback, so that the professional judgment of the reviewers can be continuously transformed into system knowledge, forming a closed loop of "review-feedback-optimization", realizing the self-learning and self-optimization of the example library, thereby continuously improving the intelligence level of the system and achieving the effect of sustainable improvement in review effectiveness after long-term use.

[0075] This embodiment provides an intelligent protocol review method, which includes: obtaining the protocol to be reviewed corresponding to the current collaborative review task; selecting guiding examples from a preset example set based on the target task information corresponding to the protocol to be reviewed; inputting the protocol to be reviewed and the guiding examples into a preset protocol review model to obtain an initial review result; obtaining user feedback information on the initial review result, and updating the preset example set according to the feedback information, so that the preset protocol review model generates a final review result based on the updated preset example set.

[0076] This embodiment first dynamically filters associated guiding examples from a preset example set based on the target task information corresponding to the protocol to be reviewed, and then updates the preset example library of the large model in a closed loop based on the user's feedback on the initial review results. This improves the dynamic adaptability of the preset example set of the large model to the collaborative review task, and thus improves the reliability of the protocol review results generated by the large model iteratively based on the updated preset example set.

[0077] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description, and will not be repeated hereafter.

[0078] It is easy to understand that existing intelligent protocol review schemes based on Few-Shot samples often use fixed examples, which can easily lead to a mismatch between the examples and real-time tasks, and large models are prone to illusions. To solve this problem, based on the first embodiment, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the smart protocol review method of this application. In this embodiment, step S20 includes steps S21 to S23: Step S21: Based on the target task information corresponding to the protocol to be reviewed, select candidate examples from the preset example set that have related attributes to the target task information; Understandably, the aforementioned association attributes can characterize the relevance between the example and the current collaborative review task. In the collaborative review scenario involved in this embodiment, it can refer to the degree of overlap between the sensitive information (and / or business scenario) contained in the example and the sensitive information (and / or business scenario) required by the current task. At this time, candidate examples can refer to all examples selected from the preset example set that have association attributes with the current task.

[0079] For example, the review device can extract the field names of sensitive information and / or business scenarios required for the current collaborative review task from the target task information to generate a list of fields to be matched, such as ["Name", "Mobile Number", "ID Type and ID Number"]. Then, it can retrieve all examples containing these field names from a preset example set, and iterate through the "Required Sensitive Information" field of each example to see if it contains at least one field from the list of fields to be matched. Finally, it can mark the examples containing the corresponding fields as candidate examples. Therefore, this embodiment can dynamically filter relevant examples based on the sensitive information of the current task, greatly improving the adaptability of the examples and guiding the large model to output more accurate results.

[0080] Step S22: Sort the candidate examples according to the credibility parameters of each candidate example to obtain the example sorting result; Step S23: Determine a guiding example from the candidate examples based on the example sorting results.

[0081] It is important to understand that the aforementioned credibility parameter is a quantitative indicator for measuring the quality and reliability of examples, with a value range of [-1, 1]. A higher value indicates a more credible example. In this case, the example ranking result can refer to a priority list generated by arranging candidate examples from highest to lowest credibility parameter and / or time proximity. That is, the review device can sort candidate examples in descending order of credibility parameter, and those with the same credibility are sorted a second time in descending order of addition time. Therefore, this embodiment, ranking by credibility, ensures that the most reliable examples are used first, further reducing the risk of illusion.

[0082] At this point, the review device can select the top K examples (K=3~5, which can be adaptively adjusted according to the context length of the preset protocol review model) from the candidate examples in the example sorting results as guiding examples. If the number of candidate examples is less than the standard number, all candidate examples can be selected, and general examples can be added to expand to the preset number. Therefore, this embodiment can adjust the value of K according to the context window size of the LLM to provide the most effective guiding information within a limited number of tokens.

[0083] It should be understood that the existing fixed-step example update method cannot reflect good data characteristics, and is prone to situations where low-quality examples can quickly improve credibility through a few positive reviews, while high-quality examples can be quickly eliminated due to a single accidental negative review. Therefore, in one possible implementation, in this embodiment, step S40 may include steps S41~S43: Step S41: Obtain the credibility statistical features corresponding to all examples in the preset example set; It is easy to understand that the aforementioned credibility statistical feature can refer to the global statistics corresponding to the credibility parameters of all examples in the preset example set. For example, the arithmetic mean or variance corresponding to the credibility of all examples can be represented as μ_score. This credibility statistical feature can be used as a benchmark reference value for subsequent offset calculation, measuring the overall credibility level of the entire preset example set.

[0084] For example, the formula for calculating the confidence statistical feature can be expressed as: μ_score=Σs_i / n, where s_i is the confidence parameter of the i-th example, and n is the total number of valid candidate examples in the preset example set.

[0085] At this point, the auditing equipment can measure the overall credibility level of the preset example set based on the credibility statistical characteristics, and determine the iterative optimization strategy for the preset example set based on the credibility statistical characteristics.

[0086] Step S42: Obtain the offset between the current credibility parameter corresponding to the guidance example and the credibility statistical feature; At this point, the aforementioned current credibility parameter can refer to the credibility value s_j stored in the guiding example that participated in generating the initial review result during this protocol review process before executing the update operation. The value range can be limited to the interval [-1, 1]. The higher the score of the current credibility parameter, the more stable the historical review performance of the corresponding candidate example and the higher its reference value.

[0087] Understandably, the aforementioned offset can refer to the normalized deviation between the current credibility parameter of a single pilot example and the average value of the credibility statistical benchmark. In this embodiment, this offset can be used to quantify the degree of difference in credibility level between candidate examples and the overall example set, and to determine the basic size of the credibility update magnitude in the preset example set.

[0088] It's easy to understand that the offset d_j can refer to the normalized deviation between the current credibility of a single bootstrap example and the average credibility statistical benchmark. In this case, the credibility calculation formula for the j-th example can be: d_j = 0.5 × |s_j - μ_score|, where the value range of d_j is fixed at [0, 1). The coefficient 0.5 is used to normalize the absolute distance to the [0, 1) interval, preventing the update step size from exceeding a reasonable range. For example, if the global average μ_score = 0.6, and the current s_j of a bootstrap example is 0.9, then d_j = 0.5 × |0.9 - 0.6| = 0.15; if another bootstrap example has s_j = 0.2, then d_j = 0.5 × |0.2 - 0.6| = 0.2.

[0089] Step S43: Determine the real-time update step size based on the feedback information and the offset, and update the credibility parameter of the guiding example in the preset example set based on the real-time update step size; The direction of the feedback information determines the adjustment direction of the real-time update step size, and the offset determines the magnitude of the real-time update step size.

[0090] It should be noted that the direction of the aforementioned feedback information can refer to the evaluation attribute corresponding to the reviewer's evaluation of the initial review result. This can include three categories: positive evaluation (such as adoption), negative evaluation (such as non-adoption), and no evaluation. Therefore, in this embodiment, the direction of the feedback information can directly determine whether the example's credibility increases or decreases. A positive evaluation corresponds to an increase in credibility (which can be represented as g=+1), a negative evaluation corresponds to a decrease in credibility (e.g., g=-2), and no evaluation means no update is performed (g=0).

[0091] It is important to understand that the offset of an example represents the degree to which the example deviates from the overall confidence level. Therefore, in this embodiment, if the offset of an example is larger, the corresponding confidence update step size can be larger; if the deviation is smaller, the confidence update step size can be smaller.

[0092] For example, when the feedback information is in a positive direction (adoption), g=1, and the formula for calculating the real-time update step size is: step=(1-t)×d_j+t, and the step size increases as d_j increases. Therefore, in this embodiment, when performing a positive update on the preset example set, low-confidence examples are improved faster, and the positive step size gradually decreases as the confidence level increases (confidence establishment is slow). Here, t is the preset minimum update step size (usually taken as 0.1). At this time, the value range of step is fixed in (-1, -t] or [t, 1) to prevent the update step size from approaching zero when the offset is too small, causing the example confidence level to stagnate for a long time.

[0093] When the feedback information is negative (not adopted), g=-1. Correspondingly, the formula for real-time update step size can be: step=-[(1-t)×d_j+t]. The absolute value of the step size increases as d_j increases. Therefore, as long as there is a low credibility example sample with negative user feedback, its credibility can drop significantly (distrust is quickly destroyed).

[0094] Finally, the auditing device can update the credibility parameter of the guide examples in the preset example set based on the real-time update step size. The update formula can be: s_j'=s_j+step. Furthermore, to ensure the legality of the credibility, the updated credibility can be constrained to the legal range of [-1, 1] by the formula: s_j'=min(max(s_j',-1), 1).

[0095] Understandably, after the aforementioned continuous updates, the credibility of high-quality examples in the preset example set will stabilize in the range of 0.8-1.0, while low-quality examples will quickly drop below -0.5 and be eliminated. Examples in the middle range are those awaiting verification, facilitating accurate system filtering. Therefore, this embodiment updates the step size in real time based on the offset and feedback information direction. This allows high-credibility examples in the preset example set to require multiple positive feedbacks to reach a perfect score, while a single negative feedback can significantly reduce their credibility. This effectively simulates the human judgment logic of "credibility is difficult to build and easy to lose," and improves the matching accuracy between examples and reality.

[0096] In summary, this embodiment updates example credibility based on real-time step sizes, rapidly eliminating low-quality examples from the preset example library while gradually accumulating high-quality examples. This effectively improves the overall quality of the example library and consequently enhances the accuracy of protocol verification for large models. Furthermore, a single, accidental incorrect evaluation will not fatally impact high-credibility examples, nor will it allow low-quality examples to infiltrate the high-credibility queue through accidental correct evaluations. Therefore, this embodiment ensures the stability and reliability of the preset example set.

[0097] This embodiment discloses a method for selecting candidate examples with associated attributes to the target task information from a preset example set based on the target task information corresponding to the protocol to be reviewed; sorting the candidate examples according to the credibility parameters of each candidate example to obtain an example sorting result; and determining a guiding example from the candidate examples based on the example sorting result. The method involves obtaining credibility statistical features corresponding to all examples in the preset example set; obtaining the offset between the current credibility parameter of the guiding example and the credibility statistical features; determining a real-time update step size based on the feedback information and the offset; and updating the credibility parameter of the guiding example in the preset example set based on the real-time update step size. The direction of the feedback information determines the adjustment direction of the real-time update step size, and the offset determines the magnitude of the real-time update step size.

[0098] This embodiment introduces offset and real-time step size to dynamically update credibility, so that the establishment of credible examples requires multiple positive accumulations and untrustworthy examples can be quickly downgraded, which is more in line with the essential characteristics of the concept of "credibility" and improves the overall reliability of the example set.

[0099] For example, to help understand the technical concept or principle of the smart protocol review method after combining this embodiment with the above-described Embodiments 1 and 2, please refer to Figure 5 , Figure 5 The following is a flowchart illustrating the optimized process of the smart protocol review method in this application: like Figure 5 As shown, in the implementation of existing intelligent protocol review methods based on traditional Few-Shot, the large language model can only rely on information based on the target task (i.e., Figure 5 (required sensitive information) and the agreement to be audited (i.e. Figure 5 The system prompt word generated by the user authorization agreement in the document (i.e.) Figure 5 The system Prompt and fixed Few-Shot samples are used for auditing. However, after the Prompt is put into production, the LLM illusion may occur, and the reliability of the audit results is not high.

[0100] To address this issue, this application proposes a smart protocol review method based on a dynamic Few-Shot enhancement mechanism. This method calculates the credibility of Few-Shot samples based on user evaluations and adoption records, thereby dynamically selecting suitable, reliable, and fresh samples to embed in the Prompt, thus improving the quality of the smart protocol review results.

[0101] At this time, as Figure 5 As shown, this application can first initialize a preset sample set (i.e. Figure 5The set of Few-Shot samples can be used to associate each sample with user input using techniques such as intent recognition and RAG (Retrieval-Augmented Generation). For example, in a collaborative review scenario, samples can be associated with sensitive information in a user-submitted sensitive information application, facilitating subsequent initial screening of Few-Shot samples.

[0102] Then, after the target task information is input, in the Few-Shot enhancement phase (i.e. Figure 5 Few-Shot Prompting can be used to select from a pre-defined set of examples (i.e., ...) based on the target task information. Figure 5 Candidate examples are selected from the set of Few-Shot examples (i.e., Figure 5 (Few-Shot sample candidates), and then filter out the leading examples ranked in the TopK based on sample credibility (i.e., Figure 5 (See the Few-Shot example in the document). Finally, the preset protocol review model in the review device can automatically generate an initial collaborative review result by comparing the sensitive information requiring authorization in the protocol to be reviewed with the registered target task information, based on the system prompts generated according to the target task information and the protocol to be reviewed, as well as the dynamically filtered guidance examples.

[0103] Meanwhile, regarding the current resource limitations of LLM, such as Figure 5 As shown, this application proposes an intelligent collaborative review architecture optimization scheme that includes a reflection strategy, specifically adding a reflection strategy to dynamically update the Few-Shot credibility of the intelligent collaborative review scheme. In this case, reviewers can evaluate and adopt the collaborative review results output by the preset protocol review model, generating feedback information (1 for positive feedback or adoption, -1 for negative feedback, and 0 for no feedback by default). Based on the feedback information, the credibility score of each example in the preset example set is used for iterative correction and output, improving usability. Simultaneously, if the feedback information is a positive evaluation, the review device can automatically convert the collaborative review result into a sample and add it to the preset example set. Furthermore, the review device can automatically eliminate examples in the preset example set based on the total number of examples in the preset example set and the example addition time. Therefore, this application can continuously provide AI collaborative review services to users by repeating the above protocol review and example update process. During the service, the Few-Shot examples and their corresponding credibility scores are continuously updated based on user feedback.

[0104] Understandably, this application can also add web crawlers and OCR tools to the collaborative review agent to support the preset protocol review model to automatically extract content and clean data of protocols provided in the form of links, and add corresponding processes for cutting and summarizing long protocols, thus broadening the application scenarios of the collaborative review agent.

[0105] Furthermore, in practical implementations, detecting complex LLM output content and long token input / output will affect the success rate and response time of LLM applications. Therefore, the more complex the LLM application scenario, the more cautious one should be in adopting conversational instant response solutions. While the Reflection strategy described in this application improves the usability of LLM, it sacrifices response time.

[0106] To ensure user experience, this application can optimize the use of the preset protocol review model through decoupling. Specifically, the application can change the LLM call executed during review to be executed asynchronously during application. This modification avoids the problems of slow response and repetitive execution in the protocol review function, adapting to the current situation where high concurrency and high-performance LLM resources cannot be supported in practical applications.

[0107] For ease of understanding, please refer to Figure 6 Provide an explanation. Figure 6 This is a comparative diagram illustrating the optimization of model invocation in the smart protocol review method of this application. For example... Figure 6 As shown, before the decoupling transformation, both the protocol review operation and multi-level feedback were located on the review side.

[0108] The management platform MySQL (My Structured Query Language) pushes the applicant's submitted agreement for review via a mini-program (i.e., Figure 6 Custom protocols in the context) and target task information (i.e. Figure 6 The required sensitive information) is sent to the preset protocol audit model (i.e. Figure 6 After the collaborative review agent (Agent) is established, the protocol review operation of the preset protocol review model and the multi-level feedback operation of the multi-level reviewers on the preset protocol review are carried out synchronously. In a single review operation, the preset protocol review model needs to receive multiple calls at the same time, and the review side can only achieve a response time of seconds.

[0109] After the decoupling transformation, the protocol review operation resides on the application side, while multi-level feedback resides on the review side. In this new model, the protocol review operation of the pre-defined protocol review model proceeds first. That is, the pre-defined protocol review model can immediately execute the protocol review upon receiving the protocol to be reviewed and the target task information, generating the corresponding collaborative review result. Then, when the reviewer needs to determine the collaborative review result of the protocol to be reviewed, the collaborative review result is directly pushed to the multi-level reviewers through the management platform, and the management platform then feeds back the feedback information to the pre-defined protocol review model for sample updates. Because the review operation is decoupled from the Agent call, a single review operation only requires a single call to the LLM, enabling millisecond-level response on the review side.

[0110] Furthermore, in this application, multiple review results output by the Generator can be sent to the Reflector for evaluation, and the best result can be selected as the final output. In specific implementations, this application can further set a strategy of concurrently executing the Generator multiple times in one iteration (multiple threads calling the LLM interface on the same server), obtaining multiple review results concurrently, thereby further reducing the impact of occasional errors in the LLM on the overall process.

[0111] In summary, this application can further decouple the preset protocol review model from the review process to alleviate the problem that LLM resources do not support high concurrency and high performance.

[0112] In summary, this application can further decouple the LLM inference task of protocol review from the review operation time to asynchronous background execution at the time of application submission, eliminating the waiting time in the foreground and adapting to the current deployment status of limited LLM resource concurrency. At the same time, in a single iteration of Reflection, multiple review results are generated by dual concurrent Generators using multiple threads, and the best result is uniformly selected and output by Reflector. The random generation characteristics of large models provide natural fault tolerance and fallback. Without increasing hardware computing power or modifying the model base, the availability, stability and front-end interaction experience of the protocol review service are significantly improved.

[0113] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the smart protocol review method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0114] This application also provides an intelligent protocol review device; please refer to [reference needed]. Figure 7 , Figure 7 This is a schematic diagram of the module structure of the intelligent protocol review device according to an embodiment of this application. In this embodiment, the intelligent protocol review device includes: The document acquisition module T1 is used to acquire the agreement to be reviewed corresponding to the current collaborative review task. Example filtering module T2 is used to filter guiding examples from a preset example set based on the target task information corresponding to the protocol to be reviewed; The preliminary review module T3 is used to input the protocol to be reviewed and the guidance example into a preset protocol review model to obtain the initial review result; Example optimization module T4 is used to obtain user feedback information on the initial review result and update the preset example set according to the feedback information, so that the preset protocol review model generates the final review result based on the updated preset example set.

[0115] The intelligent protocol review device provided in this application, employing the intelligent protocol review method in the above embodiments, can solve the technical problem of how to improve the reliability of protocol review schemes based on large models. Compared with the prior art, the beneficial effects of the intelligent protocol review device provided in this application are the same as those of the intelligent protocol review method provided in the above embodiments, and other technical features in the intelligent protocol review device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0116] This application provides an intelligent protocol review device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the intelligent protocol review method in the first embodiment described above.

[0117] The following is for reference. Figure 8 This document illustrates a structural schematic diagram of a smart protocol review device suitable for implementing embodiments of this application. The smart protocol review device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The smart protocol review device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0118] like Figure 8 As shown, the intelligent protocol auditing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1002 or programs loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the intelligent protocol auditing device. The user-related data (e.g., target task information) involved in this application are all obtained with the user's permission or consent; that is, when this application is applied to specific products or technologies, user permission is required to obtain and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations, and regulatory standards of the relevant countries and regions.

[0119] For example, when it is necessary to obtain target task information, a prompt to obtain target task information can be displayed on the user's terminal. After receiving the user's confirmation operation for the target task information prompt, the terminal can obtain the user's target task information.

[0120] Furthermore, the processing unit 1001, read-only memory 1002, and random access memory 1004 in the smart protocol auditing device are interconnected via bus 1005. Input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the smart protocol auditing device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a smart protocol auditing device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented or possessed alternatively.

[0121] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment disclosed in this application includes a smart protocol review program product, which includes a smart protocol review program carried on a computer-readable medium, the smart protocol review program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the smart protocol review program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the smart protocol review program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0122] The intelligent protocol auditing device provided in this application, employing the intelligent protocol auditing method in the above embodiments, can solve the technical problem of how to improve the reliability of protocol auditing schemes based on large models. Compared with the prior art, the beneficial effects of the intelligent protocol auditing device provided in this application are the same as those of the intelligent protocol auditing method provided in the above embodiments, and other technical features in this intelligent protocol auditing device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0123] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0124] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0125] This application provides a storage medium having computer-readable program instructions (i.e., a smart protocol auditing program) stored thereon, which are used to execute the smart protocol auditing method in the above embodiments.

[0126] The storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of the storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0127] The aforementioned storage medium may be included in the smart protocol auditing device; or it may exist independently and not be assembled into the smart protocol auditing device.

[0128] The aforementioned storage medium carries one or more programs. When the aforementioned one or more programs are executed by the smart protocol auditing device, the smart protocol auditing device becomes: smart protocol auditing.

[0129] The smart protocol auditing program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the system, method, and smart protocol auditing program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0132] The readable storage medium provided in this application is a storage medium that stores computer-readable program instructions (i.e., a smart protocol auditing program) for executing the above-described smart protocol auditing method, which can solve the technical problem of how to improve the reliability of protocol auditing schemes based on large models. Compared with the prior art, the beneficial effects of the storage medium provided in this application are the same as the beneficial effects of the smart protocol auditing method provided in the above embodiments, and will not be repeated here.

[0133] The above are only some embodiments of this application and do not limit the scope of the solution of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A smart protocol auditing method, characterized in that, The method includes: Retrieve the pending agreement corresponding to the current collaborative review task; Based on the target task information corresponding to the protocol to be reviewed, guide examples are selected from a preset example set; Input the protocol to be reviewed and the guidance example into the preset protocol review model to obtain the initial review result; Obtain user feedback on the initial review result, and update the preset example set based on the feedback, so that the preset protocol review model generates the final review result based on the updated preset example set.

2. The smart protocol auditing method as described in claim 1, characterized in that, The step of selecting guiding examples from a preset set of examples based on the target task information corresponding to the protocol to be reviewed includes: Based on the target task information corresponding to the protocol to be reviewed, candidate examples with related attributes to the target task information are selected from a preset example set; The candidate examples are sorted according to the credibility parameter of each candidate example to obtain the example sorting result; Based on the sorting results of the examples, a guiding example is determined from the candidate examples.

3. The smart protocol auditing method as described in claim 1, characterized in that, The step of updating the preset example set based on the feedback information includes: Obtain the credibility statistical features corresponding to all examples in the preset example set; Obtain the offset between the current credibility parameter corresponding to the guidance example and the credibility statistical feature; The real-time update step size is determined based on the feedback information and the offset, and the credibility parameter of the guiding example in the preset example set is updated based on the real-time update step size; The direction of the feedback information determines the adjustment direction of the real-time update step size, and the offset determines the magnitude of the real-time update step size.

4. The smart protocol auditing method as described in claim 1, characterized in that, After inputting the protocol to be reviewed and the guidance example into the preset protocol review model to obtain the initial review result, the method further includes: The initial review results are scored using a validation model to obtain a collaborative review result score and optimization suggestions. Determine whether the collaborative review result score meets the quality threshold; If the collaborative review result score does not meet the quality threshold, the optimization suggestions, the agreement to be reviewed, and the guiding example are re-input into the preset agreement review model to obtain an updated review result; The updated review result is returned to the verification model as the new review result for quality scoring again until the collaborative review result score corresponding to the new review result meets the quality threshold. The new review result is then output to the user.

5. The smart protocol auditing method as described in claim 1, characterized in that, After obtaining user feedback on the initial review result and updating the preset example set based on the feedback, the process includes: When the feedback information meets the preset supplementary conditions, the agreement to be reviewed and the initial review result are combined into a new example, and the new example is added to the preset example set; If the total number of examples corresponding to the preset example set is detected to have reached the capacity limit, a preset example elimination operation is performed based on the real-time credibility parameter of each example in the preset example set.

6. The smart protocol auditing method as described in claim 1, characterized in that, The step of obtaining the pending agreement corresponding to the current collaborative review task includes: Receive the input document information corresponding to the current collaborative review task, and determine whether the input document information is in the form of a link; If so, crawl the page source code corresponding to the input document information, perform text recognition and conversion on the non-text content in the page source code, and concatenate the converted text information with the text content of the page source code to generate an agreement to be reviewed. If not, the protocol content corresponding to the input document information will be directly used as the protocol to be reviewed.

7. The smart protocol auditing method as described in claim 1, characterized in that, Before selecting guiding examples from a preset set based on the target task information corresponding to the protocol to be reviewed, the process also includes: If the amount of data in the protocol to be reviewed exceeds the processing threshold, the protocol to be reviewed will be divided into multiple document fragments in logical order. Key information is extracted from each of the document fragments input to obtain summary information corresponding to each document fragment; The summary information is then assembled in the logical order to obtain an optimized information document, which is then used as the new agreement to be reviewed.

8. An intelligent protocol review device, characterized in that, The intelligent protocol review device includes: The document acquisition module is used to acquire the agreement to be reviewed corresponding to the current collaborative review task; The example filtering module is used to filter guiding examples from a preset example set based on the target task information corresponding to the protocol to be reviewed; The preliminary review module is used to input the protocol to be reviewed and the guidance example into a preset protocol review model to obtain the initial review result; The example optimization module is used to obtain user feedback information on the initial review result and update the preset example set according to the feedback information, so that the preset protocol review model generates the final review result based on the updated preset example set.

9. An intelligent protocol verification device, characterized in that, The device includes: a memory, a processor, and a smart protocol auditing program stored in the memory and executable on the processor, the smart protocol auditing program being configured to implement the steps of the smart protocol auditing method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores an intelligent protocol auditing program, which, when executed by a processor, implements the steps of the intelligent protocol auditing method as described in any one of claims 1 to 7.

Images