Large language model-based agreement text verification method and apparatus

Abstract

Provided in the embodiments of the present description are a large language model-based agreement text verification method and apparatus. The method comprises: determining, from amongst a plurality of preset elements, a target element corresponding to a target paragraph in a target agreement text (such as an agreement text related to privacy data, etc.), any one of the plurality of elements being a question related to agreement texts and having a preset prompt template corresponding to the element; generating a target prompt on the basis of the target paragraph and the prompt template corresponding to the target element; and inputting the target prompt into a large language model, such that the large language model performs inference related to the target element, and outputs an inference result.

Description

Protocol text detection method and device based on large language model

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 18, 2023, with application number 202311206343.9 and application name “Protocol text detection method and device based on large-scale language model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of this specification belong to the field of computer technology, and in particular to a method and device for detecting protocol text based on a large language model. Background Art

[0003] Data assets refer to data resources, such as documents and electronic data, that are owned or controlled by an enterprise and can generate future economic benefits for the enterprise. Currently, enterprises can use and process data assets, such as those containing personal privacy information, and can also sign agreements related to data assets with individuals or other enterprises.

[0004] Summary of the Invention

[0005] The purpose of the present invention is to provide a protocol text detection solution based on a large language model, which can utilize the generalization ability of the large language model to realize automatic parsing of protocol text in few-sample / zero-sample scenarios, and significantly reduce the cost of protocol text parsing.

[0006] In a first aspect, the present specification provides a protocol text detection method based on a large language model, comprising: determining a target element corresponding to a target paragraph in a target protocol text from a plurality of preset elements; wherein any one of the plurality of elements is a question related to the protocol text and is preset with a prompt template corresponding to the element; generating a target prompt based on the prompt template corresponding to the target paragraph and the target element; inputting the target prompt into the large language model, causing the large language model to perform inference related to the target element and output an inference result.

[0007] The second aspect of this specification provides a protocol text detection device based on a large language model, comprising: a determination unit, configured to determine a target element corresponding to a target paragraph in a target protocol text from a plurality of preset elements; wherein any one of the plurality of elements is a question related to the protocol text and is preset with a prompt template corresponding to the element; a generation unit, configured to generate a target prompt based on the target paragraph and the prompt template corresponding to the target element; and an inference unit, configured to input the target prompt into the large language model, so that the large language model performs inference related to the target element and outputs an inference result.

[0008] A third aspect of this specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0009] A fourth aspect of this specification provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0010] A fifth aspect of this specification provides a computer program product, which, when executed in a computer, causes the computer to execute the method described in the first aspect.

[0011] In the solution provided in the embodiments of this specification, a target element corresponding to a target paragraph in a target protocol text can be determined from a plurality of preset elements; wherein any of the plurality of elements is a question related to the protocol text and is preset with a prompt template corresponding to the element. Subsequently, a target prompt can be generated based on the prompt template corresponding to the target paragraph and the target element, and the target prompt can be input into a large language model, so that the large language model performs reasoning related to the target element and outputs the reasoning result. In this way, a prompt for input into a large language model can be generated based on the prompt template corresponding to the element, and the generalization ability of the large language model can be utilized to realize automated parsing of protocol texts in few-sample / zero-sample scenarios, significantly reducing the cost of protocol text parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0013] FIG1 is a schematic diagram of an application scenario in which the embodiments of this specification can be applied;

[0014] FIG2 is a schematic diagram of a segmentation process of a target protocol text in an embodiment of this specification;

[0015] FIG3 is a flow chart of a protocol text detection method based on a large language model according to an embodiment of the present specification;

[0016] FIG4 is a schematic diagram of a target prompt in an embodiment of this specification;

[0017] FIG5 is a schematic diagram of a manual review and feedback mechanism in an embodiment of this specification;

[0018] FIG6 is a schematic diagram of a detection process of operation behavior data in an embodiment of this specification;

[0019] FIG7 is a schematic diagram of the structure of a protocol text detection device based on a large language model in an embodiment of this specification. DETAILED DESCRIPTION

[0020] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0021] As mentioned above, enterprises may use and process data assets, such as data assets related to personal privacy data information, and may also sign agreements related to data assets with individuals or other enterprises.

[0022] In practice, the agreement text can be tested according to actual needs. For example, the agreement text can be tested for necessary clauses to prevent the agreement text from missing certain necessary clauses.

[0023] Furthermore, data, as a fundamental national strategic resource, is the core and lifeblood of the big data industry. To regulate data generation, collection, storage, processing, analysis, and services, my country has promulgated numerous laws, regulations, and policy documents. These laws, regulations, and policy documents collectively form my country's legislative framework for data governance, jointly safeguarding network and data security, promoting the development of the big data industry, unlocking the potential of data elements, and accelerating transformations in the quality, efficiency, and driving force of economic and social development. Therefore, businesses should pay greater attention to the legal and compliant collection, processing, and use of personal information. To help businesses achieve legal compliance throughout the entire lifecycle of data assets, mitigate related legal risks, assist regulators in addressing violations, and ensure that the content of agreements complies with relevant laws and regulations, compliance testing of agreement texts is recommended.

[0024] Traditional protocol text detection methods typically rely on manually written rules or machine learning models based on annotated data. These methods have several limitations, such as difficulty in writing rules, high maintenance costs, poor generalization, and difficulty and high cost in obtaining annotated data, as well as limited coverage.

[0025] Large language models (LLMs) can refer to deep neural network models trained using large amounts of text data, capable of generating natural language text based on context. LLMs are capable of few-shot / zero-shot learning. Few-shot learning refers to using a small amount of labeled data to train a model, enabling it to adapt to new tasks or categories. Zero-shot learning refers to using additional auxiliary information to train a model without any labeled data, enabling it to recognize unseen categories.

[0026] Taking into account the powerful generalization ability of large language models in few-sample / zero-sample scenarios, the embodiments of this specification provide a protocol text detection solution based on a large language model, which can utilize the generalization ability of large language models to realize automatic parsing of protocol texts in few-sample / zero-sample scenarios, significantly reducing the cost of protocol text parsing.

[0027] Figure 1 is a schematic diagram of an application scenario in which the embodiments of this specification can be applied. In the application scenario shown in Figure 1, a protocol text detection engine 101 can be included. The protocol text detection engine 101 can be used to automatically parse the protocol text.

[0028] The protocol text detection engine 101 may store a large language model M, multiple elements related to the protocol text, and prompt templates corresponding to each of these elements. In one example, the large language model M may have hundreds of billions of parameters. The large language model M may accept customized prompt input and output the results of text parsing, i.e., the inference results. In practice, prompts can be a technique that adds additional text to the input segment to better utilize the knowledge of a pre-trained language model.

[0029] In the embodiments of this specification, a single agreement text may include but is not limited to an agreement text related to privacy data, such as a privacy authorization agreement or a privacy policy. Each of the above-mentioned multiple elements can be a question related to the agreement text. The prompt template corresponding to the element can be designed by the prompt designer based on the relevant technology of the prompt engineering. The prompt template corresponding to the element can be used to instruct reasoning related to the element. In an example, the prompt template may include but is not limited to instructions containing the element and multiple alternative answers. Furthermore, the prompt template may also include at least one of the following: an example, a slot mark for characterizing a paragraph, etc. This example can be used as a reference for the large language model M, and can guide the large language model M to execute the instruction more correctly.

[0030] It should be noted that, in the scenario where compliance testing of the protocol text is required, the various elements related to the protocol text stored in the protocol text testing engine 101 may be compliance elements. It is understood that compliance elements are elements used for compliance testing.

[0031] The above-mentioned elements stored in the protocol text detection engine 101 may be designed by industry experts in accordance with laws, regulations and relevant provisions. In one example, the embodiments of this specification may involve several privacy data usage scenarios, and the above-mentioned elements may include multiple elements designed for the several privacy data usage scenarios. A single privacy data usage scenario may, for example, include any one of the following: a privacy data collection scenario, a privacy data storage scenario, a privacy data sharing scenario, and the like. Taking the privacy data collection scenario as an example, when it is necessary to perform compliance detection on the protocol text in the privacy data collection scenario, the multiple compliance elements related to the protocol text designed for the privacy data collection scenario may include, for example, whether it is stated that the personal name is collected, whether it is stated that the personal identity identifier is collected, whether there is an unreasonable disclaimer, and so on.

[0032] In one embodiment, each alternative answer in the aforementioned instruction can correspond to a label respectively. In the case where a single prompt template includes examples, instructions, and slot marks for characterizing paragraphs, taking the compliance element "Is there an unreasonable disclaimer" as an example, the prompt template P corresponding to the compliance element can be, for example, "Common unreasonable disclaimer wordings include: This platform does not assume any responsibility, users independently bear any responsibility, etc. Please use this to determine whether there is an unreasonable disclaimer in the following paragraph, and only need to answer "include" or "do not include". [Corresponding paragraph]". Among them, "Common unreasonable disclaimer wordings include: This platform does not assume any responsibility, users independently bear any responsibility, etc." is an example, "Please use this to determine whether there is an unreasonable disclaimer in the following paragraph, and only need to answer "include" or "do not include"" is an instruction, and "include" and "do not include" in the instruction are both alternative answers, and "[Corresponding paragraph]" is a slot mark for characterizing a paragraph. The label corresponding to the alternative answer "include" can be, for example, "yes", and the label corresponding to the alternative answer "does not include" can be, for example, "no". It should be understood that the examples, instructions, and labels corresponding to the various alternative answers in the prompt template can be set according to actual needs and are not specifically limited here.

[0033] Before detecting the target protocol text, the protocol text detection engine 101 or other engines may segment the target protocol text to obtain target paragraphs in the target protocol text.

[0034] When segmenting the target protocol text, the target protocol text can be structurally parsed using regular expressions and context inference technology to parse out the main sections and paragraphs of the target protocol text and the corresponding section titles.

[0035] Specifically, refer to FIG2 , which is a schematic diagram of the segmentation processing process of the target protocol text in an embodiment of this specification.

[0036] As shown in FIG2 , in step S201 , a preset regular expression for detecting multiple digital serial number identifiers is used to detect the digital serial number identifier of the target protocol text, and the detected digital serial number identifier and the title text corresponding to the digital serial number identifier are extracted from the target protocol text.

[0037] Among them, the above regular expression can be used to detect various digital serial number identifiers, such as "1.", "1)", "(one)", etc. When detecting digital serial number identifiers in the target protocol text, the target protocol text can be split by sentences or line breaks first to process the text content line by line. After obtaining each sub-text separated, the above regular expression can be used to detect digital serial number identifiers for each of these sub-texts respectively, and extract the detected digital serial number identifiers and the title text corresponding to the digital serial number identifier.

[0038] In step S203, classify the extracted digital serial number identifiers.

[0039] Specifically, the extracted digital serial number identifiers can be classified according to the type of digital serial number identifier. For example, "1.", "2." belong to one category, and "(two)", "(three)" belong to one category.

[0040] In step S205, based on the order of appearance of each category of digital serial number identifiers in the target protocol text, determine the levels corresponding to each of the extracted title texts respectively.

[0041] Among them, a single level can be, for example, first level, second level or third level, etc. Title texts of different levels can have the following characteristics: second-level title texts appear after first-level title texts, third-level title texts appear after second-level title texts, and first-level title texts may appear after second- / third-level title texts.

[0042] In step S207, perform paragraphing processing on the target protocol text based on the levels corresponding to each of the title texts respectively.

[0043] As an example, for any two adjacent first-level title texts in the target protocol text, the first-level title text with the earlier order of appearance and other texts located between the two first-level title texts can form a target paragraph.

[0044] The solution provided by the embodiment corresponding to Figure 2 is efficient and flexible. Using regular expressions to automatically match and parse digital serial number identifiers can adapt to changes in various text formats and styles, greatly reducing the time and cost of manual parsing. Through the analysis of the context relationship of digital serial number identifiers, the correctness of each level of title can be ensured. In addition, the scalability of this solution can be increased by modifying the regular expression, and it can adapt to more text formats.

[0045] Figure 3 is a flowchart of a protocol text detection method based on a large language model in an embodiment of this specification. This method can be executed by a protocol text detection engine 101.

[0046] As shown in Figure 3, first, in step S301, the target element corresponding to the target paragraph in the target agreement text is determined from multiple preset elements; wherein any element among the multiple elements is a question related to the agreement text, and a prompt template corresponding to the element is preset.

[0047] It should be noted that, when compliance testing of the target protocol text is required, the aforementioned multiple elements may be multiple compliance elements. In one example, the target protocol text may belong to one of several pre-defined privacy data usage scenarios, and the aforementioned multiple elements may be configured for that scenario. A single privacy data usage scenario may be any of the following: privacy data collection scenarios, privacy data transmission scenarios, privacy data storage scenarios, privacy data sharing scenarios, etc.

[0048] The prompt template corresponding to an element can be used to instruct reasoning related to the element. In one example, the prompt template may include, but is not limited to, instructions for the element and multiple alternative answers. Furthermore, the prompt template may also include at least one of the following: an example, a slot marker for representing a paragraph, etc. For a detailed explanation of the prompt template, please refer to the relevant description above and will not be repeated here.

[0049] In one embodiment, the above-mentioned multiple elements may correspond to keywords respectively. The keywords may be included in the corresponding elements. Taking the compliance element "whether there is an unreasonable disclaimer" as an example, the corresponding keyword may be, for example, "disclaimer." Taking the compliance element "whether it is stated that personal names are collected" as an example, the corresponding keyword may be, for example, "personal name." For an element among the above-mentioned multiple elements, if the keyword corresponding to the element is included in the paragraph content of the target paragraph, then the element may be determined as the target element corresponding to the target paragraph.

[0050] In step S303, a target prompt is generated based on the prompt template corresponding to the target paragraph and the target element.

[0051] Specifically, the target paragraph and target element corresponding prompt templates can be spliced ​​together, for example, by splicing the prompt template to the beginning or end of the target paragraph, and using the spliced ​​result as the target prompt. Alternatively, if the prompt template corresponding to the target element includes a slot marker for representing a paragraph, the target paragraph content can be written into the prompt template at the location of the slot marker to obtain the target prompt.

[0052] Taking a single prompt template including examples, instructions and slot marks for representing paragraphs as an example, assuming that the paragraph content of the target paragraph is "Article 6 Disclaimer You understand and agree that ** Bank One Account will not be responsible for the following circumstances (including but not limited to): 1. ** Bank One Account does not provide any form of guarantee for this service, including but not limited to whether this service meets your needs, and that this service is not interrupted, provided in a timely manner or free from errors. 2. The service content and quality provided by ** Bank One Account's partners are the responsibility of the partners themselves. 3. ** Bank One Account does not guarantee the accuracy and completeness of external links set up to provide convenience to members, and ** Bank One Account does not assume any responsibility for the content on any web pages that such external links point to that are not actually controlled by ** Bank One Account. 4. ** Bank One Account is not responsible for any changes, interruptions or terminations of services caused by Article 4 of this Agreement. Responsibility. 5. **Bank One Account does not assume any responsibility for any losses caused by the failure to correctly execute the **Bank One Account member operation instructions submitted by you under the following circumstances: 1. The instruction information is unclear, garbled or incomplete; 2. The products or services you own are invalid or terminated; 3. Other situations where **Bank One Account is not at fault. 6. For losses caused by computer viruses, Trojans or other malicious programs, or hacker attacks. 7. You must use your account safely and keep your **Bank One Account username and password properly. **Bank One Account is not responsible for any losses caused by your improper custody. "The target element corresponding to the target paragraph is "Whether there is an unreasonable disclaimer". This target element corresponds to the prompt template P as mentioned above. The paragraph content of the target paragraph is written into the slot mark "[Corresponding paragraph]" in the prompt template P, and the target prompt as shown in Figure 4 can be obtained. Figure 4 is a schematic diagram of the target prompt in the embodiment of this specification.

[0053] In step S305, the target word prompt is input into the large language model M, so that the large language model M performs inference related to the target element and outputs the inference result.

[0054] When a single prompt template includes the aforementioned instructions, the inference result output by the large language model M may include the answer A1 selected from the various alternative answers included in the instructions in the target prompt. Furthermore, the inference result may also include content within the target paragraph that serves as the inference basis for answer A1.

[0055] Taking the target prompt shown in FIG4 as an example, the target prompt is input into the large language model M. The large language model M can combine the example in the target prompt (“Common unreasonable disclaimer wording includes: This platform does not assume any responsibility, users independently bear any responsibility, etc.”) and execute the instruction in the target prompt (“Please judge whether there is an unreasonable disclaimer in the following paragraph based on this. Simply answer “include” or “does not include””). The large language model M infers whether there is an unreasonable disclaimer in the content of the target paragraph and outputs an inference result, which includes answer A1 selected from the alternative answers “include” and “does not include”. It should be understood that answer A1 is “include” or “does not include”. Assuming that the answer to A1 is "include", the content in the target paragraph that serves as the reasoning basis for answer A1 may include, for example, "3. **Bank One Account does not guarantee the accuracy and completeness of external links set up to provide convenience to members. At the same time, **Bank One Account does not assume any responsibility for the content on any web pages that are not actually controlled by **Bank One Account and that such external links point to. 5. **Bank One Account does not assume any responsibility for any losses caused by failure to correctly execute the **Bank One Account member operation instructions you submitted under the following circumstances."

[0056] The solution provided by the embodiment corresponding to FIG3 can determine the target element corresponding to the target paragraph in the target agreement text from a plurality of preset elements; wherein any of the plurality of elements is a question related to the agreement text and is preset with a prompt template corresponding to the element. Afterwards, a target prompt can be generated based on the prompt template corresponding to the target paragraph and the target element, and the target prompt can be input into a large language model so that the large language model performs reasoning related to the target element and outputs the reasoning result. In this way, a prompt for input into a large language model can be generated based on the prompt template corresponding to the element, and the generalization ability of the large language model can be utilized to realize the automated parsing of the agreement text in a few-sample / zero-sample scenario, significantly reducing the cost of parsing the agreement text.

[0057] In one embodiment, to ensure the reliability of the inference results of the large language model M, after obtaining the inference results output by the large language model M, the inference results can be manually reviewed. For example, a target prompt and inference results can be provided to a result reviewer, allowing the reviewer to review the correctness of the answer A1 in the inference result. Based on this, the application scenario shown in Figure 1 can also include a result review device 102 used by the result reviewer, and the protocol text detection engine 101 can communicate with the result review device 102.

[0058] In addition, to improve the inference accuracy of the large language model M, when answer A1 fails manual review, information feedback can be provided to the prompt designer to prompt the prompt designer that the example in the prompt template corresponding to the target element is an incorrect example, thereby allowing the prompt designer to optimize the prompt template. Based on this, the application scenario shown in Figure 1 can also include a prompt design device 103 used by the prompt designer, and the protocol text detection engine 101 can communicate with the prompt design device 103.

[0059] Next, the manual review and feedback mechanism in the embodiment of this specification will be described with reference to Figure 5. Figure 5 is a schematic diagram of the manual review and feedback mechanism in the embodiment of this specification.

[0060] As shown in FIG5 , in step S501 , the protocol text detection engine 101 sends the target prompt and the inference result to the result review device 102 , so that the result reviewer reviews the correctness of the answer A1 in the inference result.

[0061] It should be noted that the result review device 102 can display the received target prompt and reasoning result to the result reviewer, so that the result reviewer can review the correctness of the answer A1 in the reasoning result based on the target prompt. In one example, in order to facilitate manual review, the reasoning result can also include content that serves as the reasoning basis for the answer A1, and the result reviewer can review the correctness of the answer A1 based on the target prompt and the content. After reviewing the correctness of the answer A1, the result reviewer can submit the review result to the result review device 102. The review result is either passed or failed. Afterwards, the result review device 102 can execute step S503.

[0062] In one embodiment, when the audit result is a failure, the result reviewer can modify the incorrect answer A1 to the correct answer A2 and include answer A2 in the audit result. It is understood that answer A2 is selected by the result reviewer from the various alternative answers included in the instructions in the target prompt.

[0063] In step S503, the result review device 102 sends the review result to the protocol text detection engine 101, which is submitted by the result reviewer after reviewing the correctness of the answer A1 in the reasoning result.

[0064] In step S505 , the protocol text detection engine 101 determines whether the audit result is passed.

[0065] As an example, the audit result may include a mark indicating whether the audit is passed or not. The protocol text detection engine 101 can determine whether the audit result is passed by identifying the mark. If the audit result is not passed, the protocol text detection engine 101 can execute step S507. In addition, regardless of whether the audit result is passed or not, the protocol text detection engine 101 can execute step S509.

[0066] In step S507, in response to the audit result being a failed audit, the protocol text detection engine 101 sends feedback information to the prompt design device 103 indicating that the example in the prompt template corresponding to the target compliance element is an incorrect example, so that the prompt designer can optimize the prompt template.

[0067] It should be noted that after receiving the feedback information, the prompt design device 103 can display the feedback information to the prompt designer, so that the prompt designer can optimize the prompt template corresponding to the target element, such as optimizing the examples in the prompt template. Taking the aforementioned prompt template T corresponding to the target element as an example, the prompt designer can, for example, modify the content after "Common unreasonable disclaimer wording includes:" in prompt template T to improve the inference accuracy of the large language model M.

[0068] In step S509, the protocol text detection engine 101 determines the target answer based on the audit result; when the audit result is passed, the target answer is answer A1; when the audit result is failed and includes answer A2 provided by the result reviewer, the target answer is answer A2.

[0069] In step S511 , the protocol text detection engine 101 determines the label corresponding to the target answer as the target label corresponding to the target element.

[0070] According to the above description, each alternative answer included in the prompt template can correspond to a label. After determining the target answer, the protocol text detection engine 101 can map the target answer to its corresponding label and determine the label as the target label corresponding to the target element.

[0071] The solution provided by the embodiment corresponding to FIG5 can ensure the reliability of the inference results of the large language model M and improve the inference accuracy of the large language model M through manual review and feedback mechanism.

[0072] In one embodiment, the target protocol text can be used to regulate user A's operational behavior on private data. User A can be, for example, an enterprise user. The operational behavior can include, for example, any of the following: collection behavior, transmission behavior, storage behavior, sharing behavior, etc. A downstream engine (such as a decision engine) of the protocol text detection engine 101 can be used to detect the operational behavior data generated based on user A's operational behavior. The protocol text detection engine 101 determines each target element and the target label corresponding to each target element by detecting the target protocol text, which can be used as a reference for the downstream engine when detecting the operational behavior data.

[0073] Based on this, the application scenario shown in FIG1 may further include a decision engine 104 , and the protocol text detection engine 101 may provide each target element and its corresponding target label to the decision engine 104 for reference by the decision engine 104 .

[0074] In one embodiment, the protocol text detection engine 101 may also store a number of rules related to the target protocol text, and a single rule may include obligation conditions related to the above-mentioned operational behavior, and obligation requirements containing elements related to the protocol text. The element is included in the multiple elements mentioned in step S301 as described above. In an example, when the multiple elements are multiple compliance elements, the multiple rules may be referred to as multiple compliance rules, and the multiple compliance rules may be provided to the decision engine 104 for compliance detection of operational behavior data. The multiple rules may be designed by industry experts in accordance with laws, regulations and relevant provisions. In the case where the multiple elements are configured for a certain scenario among the multiple privacy data usage scenarios described above, the multiple rules may also be configured for the scenario.

[0075] In practice, when an enterprise wants to collect personal privacy data, the relevant agreement needs to make requirements on the type of data that the privacy data to be collected belongs to. Based on this, when the above-mentioned several rules are several compliance rules configured for the privacy data collection scenario, user A's operation behavior on the privacy data can be a collection behavior, and the several compliance rules may include compliance rule R1 and compliance rule R2, etc.; among them, the obligation condition in compliance rule R1 may include, for example, that the data type belongs to personal name, and the obligation requirement may include, for example, the compliance element "whether to declare the collection of personal name" and the label corresponding to the compliance element (for example, "yes"); the obligation condition in compliance rule R2 may include, for example, that the data type belongs to personal identification, and the obligation requirement may include, for example, the compliance element "whether to declare the collection of personal identification" and the label corresponding to the compliance element (for example, "yes").

[0076] The following describes the detection process of the operation behavior data in conjunction with Figure 6. Figure 6 is a schematic diagram of the detection process of the operation behavior data in the embodiment of this specification.

[0077] As shown in FIG6 , in step S601 , the protocol text detection engine 101 sends to the decision engine 104 several rules related to the target protocol text, as well as each target element determined by detecting the target protocol text and the target labels corresponding to each target element.

[0078] In step S603, for a rule among the above-mentioned rules, if the operation behavior data generated based on the operation behavior of user A on the privacy data meets the obligation condition in the rule, then the target element included in the obligation requirement of the rule is determined in each target element, and whether the operation behavior data meets the obligation requirement is judged based on the target label corresponding to the target element, and a detection result is generated based on the judgment result, and the detection result is used to indicate whether the operation behavior data passes the rule.

[0079] It should be noted that when matching operational behavior data with any of the aforementioned rules, if the operational behavior data does not satisfy the obligation conditions in that rule, then there is no need to continue matching the operational behavior data with the obligation requirements in that rule. If the operational behavior data satisfies the obligation conditions in that rule, then there is a need to continue matching the operational behavior data with the obligation requirements in that rule.

[0080] Taking compliance rule R1 as an example, let's assume that operational behavior data is generated based on user A's collection of a personal name. This operational behavior data includes, for example, a field name representing the personal name and a field value corresponding to the field name. By matching this operational behavior data with the obligation conditions in compliance rule R1, it can be determined that the operational behavior data satisfies the obligation conditions. Then, the target elements included in the obligation requirements of compliance rule R1 can be determined within each target element, such as "whether to declare the collection of personal names." Next, based on the target element "whether to declare the collection of personal names" and its corresponding target label, it can be determined whether the operational behavior data satisfies the obligation requirement, and a test result is generated based on the determination result. It can be understood that when the target label corresponding to the target element is "yes," the operational behavior data can be determined to satisfy the obligation requirement, thereby generating a test result indicating that the operational behavior data passes compliance rule R1. This test result can indicate that the target agreement text declares the collection of personal names, and that user A's collection of personal names is compliant. In addition, when the target label corresponding to the target element is "no", it can be determined that the operation behavior data does not meet the obligation requirements, thereby generating a detection result indicating that the operation behavior data does not pass the compliance rule R1. The detection result can indicate that the target agreement text does not declare the collection of personal names, and user A's collection of personal names is non-compliant.

[0081] In step S605 , the decision engine 104 generates a detection report based on the generated detection results.

[0082] The test report may include the test results. Further, the test report may include other information, such as the number of rules that the operation behavior data passes and the number of rules that it fails, the test time, etc.

[0083] In the embodiment corresponding to Figure 6 , the target tag corresponding to the target element is the tag corresponding to the target answer, which is determined based on manual review and has extremely high reliability. By providing the decision engine with several rules related to the target protocol text, as well as the target elements determined through testing the target protocol text and their corresponding target tags, for automated detection of operational behavior data, detection efficiency and accuracy can be effectively improved.

[0084] According to the description in the foregoing, the embodiment of this specification proposes a solution for automatic parsing of protocol texts based on a large language model, which can solve the problem of automatic parsing of protocols in few-sample / zero-sample scenarios. This solution can use the generalization ability of large language models to solve this problem. The pre-trained large language model (such as GPT-3.5) has been subjected to unsupervised learning on massive texts. By simply and effectively adjusting the prompt information, it can complete the task of automatic parsing of protocol texts of different types and fields, extract key information and logical relationships from the protocol text, and convert them into structured data for downstream decision-making engine reasoning and judgment. In addition, by adding manual review and feedback mechanisms, it is possible to ensure the reliability of the reasoning results and continuously improve the reasoning accuracy of the model. This solution uses elements (such as compliance elements) and rules (such as compliance rules) as a way to define decision logic, which increases configurability and scalability.

[0085] FIG7 is a schematic diagram of the structure of a protocol text detection device based on a large language model in an embodiment of the present specification. The device can be applied to the protocol text detection engine 101 as described above, for example. The device can execute the methods shown in FIG2 , FIG3 , FIG5 , and FIG6 , respectively. The device may include: a determination unit 701, configured to determine a target element corresponding to a target paragraph in a target protocol text from a plurality of preset elements; wherein any element among the plurality of elements is a question related to the protocol text and is preset with a prompt template corresponding to the element; a generation unit 702, configured to generate a target prompt based on the prompt template corresponding to the target paragraph and the target element; and an inference unit 703, configured to input the target prompt into a large language model, so that the large language model performs inference related to the target element and outputs an inference result.

[0086] The embodiments of this specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the methods shown in Figures 2, 3, 5, and 6, respectively.

[0087] The embodiments of this specification further provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the methods shown in FIG. 2 , FIG. 3 , FIG. 5 , and FIG. 6 are implemented respectively.

[0088] The embodiments of this specification also provide a computer program product, wherein when the computer program product is executed in a computer, the computer is caused to execute the methods shown in FIG. 2 , FIG. 3 , FIG. 5 , and FIG. 6 , respectively.

[0089] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0090] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0091] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the future development of computer technology, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0092] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any particular order.

[0093] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0094] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0095] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0097] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0098] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0099] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0100] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0102] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0103] The foregoing description is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. Those skilled in the art will appreciate that various modifications and variations of one or more embodiments of this specification are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification are intended to be included within the scope of the claims.

Claims

1. A protocol text detection method based on a large language model, comprising: Determining a target element corresponding to a target paragraph in a target agreement text from a plurality of preset elements; wherein any one of the plurality of elements is a question related to the agreement text, and a prompting template corresponding to the element is preset; Generate a target prompt based on the prompt template corresponding to the target paragraph and the target element; The target word prompt is input into the large language model, so that the large language model performs reasoning related to the target element and outputs the reasoning result.

2. The method according to claim 1, wherein: The multiple elements are multiple compliance elements.

3. The method according to claim 1, wherein: Before determining the target element corresponding to the target paragraph in the target protocol text from the preset multiple elements, the method further includes: The target protocol text is segmented to obtain target paragraphs of the target protocol text.

4. The method according to claim 3, wherein: The segmenting of the target protocol text includes: Using a preset regular expression for detecting multiple digital serial number identifiers, performing digital serial number identifier detection on the target protocol text, and extracting the detected digital serial number identifier and the title text corresponding to the digital serial number identifier from the target protocol text; Classify the extracted digital serial number identifiers; Determine the level corresponding to each extracted title text based on the order of appearance of each type of digital serial number identifier in the target protocol text; The target protocol text is segmented based on the levels corresponding to the various title texts.

5. The method according to claim 1, wherein: The multiple elements correspond to keywords respectively; as well as The step of determining the target element corresponding to the target paragraph in the target protocol text from the preset multiple elements includes: For an element among the multiple elements, if a keyword corresponding to the element is included in the paragraph content of the target paragraph, the element is determined as a target element corresponding to the target paragraph.

6. The method according to any one of claims 1 to 5, wherein: The prompt template corresponding to any of the elements includes an example, an instruction containing the element and a plurality of alternative answers, and a slot mark for representing a paragraph; as well as The generating a target word prompt based on the word prompt template corresponding to the target paragraph and the target element includes: The paragraph content of the target paragraph is written into the position of the slot mark in the prompt template corresponding to the target element to obtain the target prompt.

7. The method according to claim 6, wherein: The inference result includes a first answer selected from each alternative answer included in the instruction in the target prompt; as well as The method further comprises: Providing the target prompt and the inference result to a first user, so that the first user can review the correctness of the first answer; Receive the review result submitted by the first user.

8. The method according to claim 7, wherein: The prompt templates corresponding to the plurality of elements are configured by the second user in the prompt design device; as well as The method further comprises: In response to the audit result being a failure to pass the audit, feedback information indicating that the example in the prompt template corresponding to the target element is an erroneous example is sent to the prompt design device, so that the second user optimizes the prompt template.

9. The method according to claim 7, wherein: Each of the alternative answers corresponds to a label; as well as The method further comprises: Determine a target answer based on the review result; wherein, when the review result is approved, the target answer is the first answer; when the review result is unapproved and includes the second answer selected by the first user from the various alternative answers, the target answer is the second answer; The label corresponding to the target answer is determined as the target label corresponding to the target element.

10. The method according to claim 9, wherein: The target agreement text is used to regulate the operation behavior of the third user on the private data, and is associated with a plurality of preset rules, wherein a single rule includes an obligation condition related to the operation behavior, and an obligation requirement including an element of the plurality of elements; as well as The method further comprises: Inputting the determined target elements, the target tags respectively corresponding to them, and the plurality of rules into a decision engine, the decision engine performing detection processing on the operation behavior data generated based on the operation behavior of the third user; the detection processing includes: For a rule among the plurality of rules, if the operation behavior data satisfies the obligation condition in the rule, then determining a target element included in the obligation requirement of the rule among the target elements, judging whether the operation behavior data satisfies the obligation requirement based on a target tag corresponding to the target element, and generating a detection result based on the judgment result, wherein the detection result is used to indicate whether the operation behavior data passes the rule; Generate a test report based on each test result generated.

11. The method according to claim 10, wherein: The target protocol text belongs to a scenario among several preset privacy data usage scenarios, and the multiple elements and the multiple rules are configured for the scenario.

12. The method according to claim 10, wherein: A single privacy data usage scenario is any one of the following: privacy data collection scenario, privacy data transmission scenario, privacy data storage scenario, and privacy data sharing scenario.

13. A protocol text detection device based on a large language model, comprising: A determination unit is configured to determine a target element corresponding to a target paragraph in a target agreement text from a plurality of preset elements; wherein any element among the plurality of elements is a question related to the agreement text, and a prompting template corresponding to the element is preset; A generating unit, configured to generate a target prompt based on the prompt template corresponding to the target paragraph and the target element; The inference unit is configured to input the target word into the large language model, so that the large language model performs inference related to the target element and outputs an inference result.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 12.

15. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 12 is implemented.

Images