Examination point knowledge base generation method and device, and examination content determination method and device

Abstract

Embodiments of the present application provide a review point knowledge base generation method and device, and a review content determination method and device, relating to the technical field of artificial intelligence. The method comprises: determining judicial interpretation data according to a legal database; the judicial interpretation data comprises multiple judicial interpretation texts; determining contract data according to industry standards; the contract data comprises multiple different types of contracts; generating a review point knowledge base according to the correlation between the contract data and the judicial interpretation data; the review point knowledge base comprises multiple review points for reviewing contracts; the review point is a review point corresponding to the judicial interpretation data related to the contract data. The scheme of the present application provides data basis support for subsequent extraction of review points in the contract to be reviewed through the generation of the review point knowledge base, solving the problem of high labor cost in extracting review points.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for generating a review point knowledge base, and a method and apparatus for determining review content. Background Technology

[0002] With the advent of the big data era, the demand for efficiently and accurately extracting and organizing key information from massive amounts of unstructured text is growing. Especially for legal contract documents, existing technologies do not focus on the definition and discovery of "review points," nor do they address how to dynamically identify review points from constantly updated laws and regulations. Therefore, when faced with rapid changes in laws and policies, systems cannot automatically capture and integrate new review dimensions, still requiring significant manual intervention to update their review logic. Existing technologies face challenges such as high labor costs, difficulties in information location, and the complexity of dynamically updating review points when developing and extracting diverse review points efficiently and accurately. Summary of the Invention

[0003] At least one embodiment of this application provides a method and apparatus for generating a review point knowledge base, and a method and apparatus for determining review content, which are used to solve the problem of high labor costs in the prior art when extracting diverse review points efficiently and accurately.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a method for generating an examination point knowledge base, including:

[0006] Based on the legal database, judicial interpretation data is determined; the judicial interpretation data includes multiple original texts of judicial interpretations.

[0007] Contract data is determined according to industry standards; the contract data includes various types of contracts.

[0008] A review point knowledge base is generated based on the correlation between the contract data and the judicial interpretation data; the review point knowledge base includes multiple review points for reviewing the contract; the review points are the review points corresponding to the judicial interpretations related to the contract data in the judicial interpretation data.

[0009] Optionally, based on the correlation between the contract data and the judicial interpretation data, a review point knowledge base is generated, including:

[0010] The contracts in the contract data are classified according to contract type to obtain classified contract data corresponding to different types of contracts.

[0011] Determine the keyword database for the categorized contract data;

[0012] Based on the correlation between the judicial interpretation data and the keyword database, the target judicial interpretation is determined;

[0013] The knowledge base for review points is determined based on the aforementioned target judicial interpretation.

[0014] Optionally, the target judicial interpretation is determined based on the relevance between the judicial interpretation data and the keyword database, including:

[0015] Each judicial interpretation in the data is broken down into multiple clause units; each clause unit includes a complete legal rule.

[0016] The multiple clause units are merged to obtain clause segments;

[0017] The relevance between the clause segments and the keyword database is determined using a large model;

[0018] Based on the relevance, target clause segments related to the keyword database are determined;

[0019] Determine the target judicial interpretation corresponding to the target clause segment.

[0020] Optionally, the review point knowledge base is determined according to the target judicial interpretation, including:

[0021] The target judicial interpretation is broken down into multiple target judicial interpretation clauses;

[0022] The target judicial interpretation clauses are processed by a large model to obtain target review point data corresponding to multiple target judicial interpretation clauses; the target review point data includes: multiple target review points, as well as the identifier, risk point, content, applicable legal information and review legal basis of each target review point;

[0023] Based on the target review point data, construct the review point knowledge base.

[0024] Secondly, embodiments of this application provide a method for determining review content, applied to the review point knowledge base generation method described above, including:

[0025] The contract to be reviewed is pre-annotated to determine the annotation information of the contract to be reviewed; the pre-annotation process includes: large language model pre-annotation and keyword pre-annotation; the annotation information includes: keywords and the position information of the keywords;

[0026] Identify the target review points in the review point knowledge base that are related to the annotation information;

[0027] Based on the target review points and the annotation information, the content to be reviewed in the contract to be reviewed is determined.

[0028] Optionally, the contract to be reviewed is pre-annotated to determine the annotation information of the contract to be reviewed, including:

[0029] The contract to be reviewed is split into multiple text segments according to logical units;

[0030] Multiple text segments were pre-annotated using a large model to obtain multiple primary keywords;

[0031] The contract to be reviewed is pre-labeled with keywords to determine the second keyword and the location information of the keyword in the contract to be reviewed; the keyword includes the first keyword and the second keyword.

[0032] Optionally, keyword pre-tags are performed on the contract to be reviewed to determine the second keyword and the position information of the keyword in the contract to be reviewed, including:

[0033] A string search algorithm is used to pre-annotate keywords in the contract to be reviewed based on a keyword database to obtain a second keyword and the location information of the keyword.

[0034] The keyword database includes keyword data corresponding to various types of contracts; the keyword location information includes the start and end positions of the keyword in the contract to be reviewed.

[0035] Optionally, the contract to be reviewed is split into multiple text segments according to logical units, including:

[0036] Based on the structural features and overall meaning of the contract to be reviewed, the contract to be reviewed is segmented to obtain the first text segment;

[0037] The first text segment is subjected to information extraction processing to obtain the plurality of text segments.

[0038] Optionally, based on the target review points and the annotation information, the content to be reviewed in the contract to be reviewed is determined, including:

[0039] When multiple keywords exist, semantic understanding and reasoning are performed on the keywords to determine the content to be reviewed between the target start position and the target end position.

[0040] Thirdly, embodiments of this application provide a knowledge base generation apparatus, including:

[0041] The first determining module is used to determine judicial interpretation data based on a legal database; the judicial interpretation data includes multiple original texts of judicial interpretations.

[0042] The second determining module is used to determine contract data according to industry standards; the contract data includes various types of contracts.

[0043] The generation module is used to generate a review point knowledge base based on the correlation between the contract data and the judicial interpretation data; the review point knowledge base includes multiple review points for reviewing the contract; the review points are the review points corresponding to the judicial interpretations related to the contract data in the judicial interpretation data.

[0044] Fourthly, embodiments of this application provide an apparatus for determining the content to be examined, comprising:

[0045] The third determining module is used to perform pre-annotation processing on the contract to be reviewed, and to determine the annotation information of the contract to be reviewed; the pre-annotation processing includes: large language model pre-annotation and keyword pre-annotation; the annotation information includes: keywords and the position information of the keywords;

[0046] The fourth determining module is used to determine the target review points in the review point knowledge base that are related to the annotation information;

[0047] The fifth determining module is used to determine the content to be reviewed in the contract to be reviewed based on the target review points and the annotation information.

[0048] Fifthly, embodiments of this application provide a network device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described in the first or second aspect.

[0049] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the method described in the first or second aspect.

[0050] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0051] Compared with existing technologies, the method and apparatus for generating a review point knowledge base and the method and apparatus for determining review content provided in this application embodiment determine judicial interpretation data through a legal database and contract data including various types of contracts according to industry standards; then, based on the correlation between the contract data and the judicial interpretation data, a review point knowledge base is generated; the contract to be reviewed is pre-annotated to determine the annotation information of the contract to be reviewed; thereby determining the target review points in the review point knowledge base corresponding to the annotation information; finally, the content to be reviewed in the contract to be reviewed is determined based on the target review points and the standard information. This solves the problem of high labor costs faced by existing technologies in efficiently and accurately extracting diverse review points. Attached Figure Description

[0052] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0053] Figure 1 This is a schematic diagram illustrating the steps of the knowledge base generation method in an embodiment of this application;

[0054] Figure 2 This is a schematic diagram illustrating the process of decomposing and preprocessing judicial interpretations according to an embodiment of this application.

[0055] Figure 3 This is a breakdown of the judicial interpretation of the embodiments of this application and a matching with the contract to be examined;

[0056] Figure 4 This is a schematic diagram illustrating the steps of the method for determining the examination content in an embodiment of this application;

[0057] Figure 5 This is a schematic diagram illustrating the steps of review point confirmation using a large language model in an embodiment of this application.

[0058] Figure 6 This is a schematic diagram of the modules of the knowledge base generation device according to an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of the module of the examination content determination device according to an embodiment of this application;

[0060] Figure 8 This is a schematic diagram of the structure of a network device according to another embodiment of this application. Detailed Implementation

[0061] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0062] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc.; an indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0063] To enable those skilled in the art to better understand the embodiments of this application, the following description is provided first:

[0064] This application relates to the field of artificial intelligence, particularly Natural Language Processing (NLP) technology, especially information extraction and text classification. With the advent of the big data era, the need for efficient and accurate extraction and organization of key information from massive amounts of unstructured text is growing. This section will outline existing NLP technologies related to this invention, including traditional methods, deep learning paradigms, recent advancements in large language models, and human-computer collaborative annotation techniques.

[0065] Text processing and information extraction fundamental technologies: Before the rise of large-scale pre-trained language models, text classification and information extraction were the two core tasks in the field of NLP, and various technical approaches were developed.

[0066] Text classification aims to categorize text content into predefined categories.

[0067] Traditional machine learning methods include feature engineering-based approaches, such as using TF-IDF, Bag-of-Words (BoW), and N-grams as text features, combined with classifiers like Support Vector Machines (SVM), Naive Bayes, Logistic Regression, and Decision Trees for text classification. Additionally, topic models such as Latent Dirichlet Allocation (LDA) and Nonnegative Matrix Factorization (NMF) are also used to discover the topic distribution of documents.

[0068] Deep learning methods: With the development of neural network technology, convolutional neural networks (CNNs, such as TextCNN and KimCNN) capture local features of text through convolutional kernels; recurrent neural networks (RNNs, such as LSTM and GRU) are suitable for processing sequential data and capturing textual context information. The attention mechanism introduced on this basis further enhances the model's ability to focus on more important parts of the text.

[0069] 2. Information extraction technology

[0070] Information extraction aims to identify and extract structured information from unstructured text.

[0071] Named Entity Recognition (NER) identifies entities in text that have specific meanings, such as names of people, places, and organizations. Methods include rule / dictionary-based approaches (predefined rules, keyword lists), statistical machine learning-based approaches (such as Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs), and deep learning-based approaches (such as BiLSTM-CRF, BiLSTM-CRF combined with pre-trained language models (PLMs), and Span-based extraction methods).

[0072] Relation extraction (RE) and event extraction: Relation extraction aims to identify semantic relationships between entities in text; event extraction focuses on identifying event trigger words, event types, and event arguments. These tasks are typically achieved through pattern matching, statistical classifiers, or deep learning-based methods (such as CNNs, RNNs, and graph neural networks (GNNs) combined with attention mechanisms).

[0073] Joint Extraction: This involves jointly modeling multiple information extraction tasks such as NER and RE to share information and improve the overall extraction effect and consistency.

[0074] 3. Text Length and Granularity Processing Strategies

[0075] When processing texts of varying lengths, the industry has developed specific strategies:

[0076] Long text processing: For texts that are lengthy and where key information may be scattered, techniques such as segmentation, attention mechanisms, or hierarchical modeling are typically needed to reduce computational load and improve the ability to capture key information. For example, in text classification tasks, segmenting long texts before aggregating information is a common strategy.

[0077] Short text information extraction: In particular, the Unified Information Extraction (UIE) paradigm focuses on the accurate extraction of shorter fields such as phrases and entities, and usually relies more on the accurate identification of text boundaries and the deep understanding of local semantics.

[0078] In recent years, the rapid development of Large Language Models (LLMs) has completely changed the landscape of the NLP field, bringing revolutionary capabilities to various tasks.

[0079] 1. The Development History of Large Language Models (PLMs / LLMs)

[0080] From early models like ELMo, GPT, and BERT, to later models such as T5, GPT-3, InstructGPT, ChatGPT, Llama, and GLM, these models have been pre-trained on massive amounts of text data, acquiring rich linguistic knowledge, world knowledge, and powerful reasoning abilities, thus becoming general-purpose language understanding and generation tools.

[0081] 2. A New Paradigm of LLMs in Information Extraction and Text Classification

[0082] LLMs, with their powerful generalization and contextual understanding capabilities, offer new solutions for information extraction and text classification tasks, significantly reducing the reliance on large amounts of labeled data.

[0083] Zero-shot / Few-shot learning: Through carefully designed prompt engineering or a small number of examples, large models can perform information extraction and text classification tasks such as NER and RE without a large amount of task-specific labeled data.

[0084] Instruction Tuning / Prompt Learning: By designing and optimizing instructions for specific tasks, large models are guided to generate information that meets the requirements or to make classification judgments, thereby further improving task performance.

[0085] Generative information extraction: LLMs can directly generate extraction results in structured formats (such as JSON, lists), rather than relying on classification or sequence labeling as in traditional extraction methods, thus simplifying the information acquisition process.

[0086] 3 Application of LLMs in Data Labeling and Augmentation

[0087] The ability to generate LLMs makes them an important auxiliary tool in data processing workflows:

[0088] Automated annotation / pre-annotation: Large models can perform preliminary annotation on a large amount of unannotated data based on instructions and context, serving as a starting point for manual annotation and significantly improving annotation efficiency.

[0089] Data Augmentation: LLMs can generate more training data or perturb existing data to increase data diversity, thereby improving the model's generalization ability.

[0090] 4 Other advanced applications of LLMs

[0091] LLMs have also demonstrated strong potential in other advanced NLP tasks, including:

[0092] Knowledge matching / question answering: It can understand the deep semantics of text, perform text similarity calculation, semantic matching, or directly answer questions.

[0093] Logical Reasoning / Summary: Possesses a certain level of logical reasoning ability, able to infer implicit information from text, and efficiently extract key information from text for summarization or conclusion.

[0094] To compensate for the shortcomings of automated annotation and reduce the cost of manual annotation, human-computer collaborative annotation technology has been widely used.

[0095] Active Learning: The algorithm actively selects the unlabeled data that is most helpful for model training through methods such as uncertainty sampling, committee query, and density-weighted sampling, and then hands it over to experts for labeling, so as to achieve better model performance with less labeling cost.

[0096] Weak supervision: Uses imprecise or incomplete labeled information (such as rules, heuristics, external knowledge bases) to train models, reducing the reliance on precise human annotations.

[0097] Human-computer collaborative annotation platforms: The industry has developed platforms such as Docano, Prodigy, and Label Studio, which provide user-friendly web interfaces for text annotation and support various annotation types such as classification, sequence labeling, and relation extraction, facilitating manual verification and correction.

[0098] Model-assisted annotation: The trained model makes preliminary predictions on unlabeled data, and then manual corrections are made based on these predictions, which greatly improves the annotation efficiency.

[0099] As described in the background section, existing technologies face challenges such as high labor costs, difficulty in information location, and complex dynamic updates of review points when efficiently and accurately extracting diverse review points. To address at least one of these issues, this application provides a method for generating a review point knowledge base and a method for determining review content, which can reduce or avoid the occurrence of the above situations.

[0100] This application provides a method and apparatus for generating an examination point knowledge base, and a method and apparatus for determining examination content. The method and apparatus are based on the same concept, and since they solve problems based on similar principles, their implementations can be mutually referenced; repeated details will not be elaborated further.

[0101] like Figure 1 As shown in the embodiment of this application, a method for generating an examination point knowledge base includes the following steps:

[0102] Step 101: Determine judicial interpretation data based on the legal database; the judicial interpretation data includes multiple original texts of judicial interpretations.

[0103] Step 102: Determine the contract data according to industry standards; the contract data includes various types of contracts.

[0104] Step 103: Generate a review point knowledge base based on the correlation between the contract data and the judicial interpretation data; the review point knowledge base includes multiple review points for reviewing the contract; the review points are the review points in the judicial interpretation data that correspond to the judicial interpretations related to the contract data.

[0105] The method for generating a review point knowledge base provided in this application involves determining judicial interpretation data from a legal database and contract data including various types of contracts based on industry standards. Then, a review point knowledge base is generated based on the correlation between the contract data and the judicial interpretation data, providing a data foundation for the subsequent extraction of review points from contracts under review.

[0106] Optionally, the legal database is an authoritative database of judicial interpretations, and the legal database includes laws and regulations that are updated in real time.

[0107] Optionally, after determining the judicial interpretation data based on the legal database, the following may also be included:

[0108] The judicial interpretation data is preprocessed by data cleaning and format conversion to obtain the judicial interpretation data in plain text format.

[0109] In this embodiment of the application, the data for the judicial interpretation is preprocessed, including:

[0110] The original texts of judicial interpretations were retrieved in batches from authoritative legal databases, initially in HTML format. To eliminate the potential interference of HTML tags and other formatting elements on subsequent text analysis, a data cleaning process was used, employing regular expression technology to remove formatting information and convert the original texts of the judicial interpretations into plain text format, as shown in Table 1. This ensured the cleanliness and usability of the text, allowing each judicial interpretation clause to enter the subsequent processing stage in a clear form.

[0111] Table 1 Comparison of Plain Text Data in the Original Data Field of Judicial Interpretations

[0112]

[0113] In this embodiment, the contract data includes data compiled based on legal business experience and industry standards; and the contract data includes contracts categorized into different systems. For example, there are lease contracts, technology contracts, etc., and each contract type is further subdivided into multiple contract subcategories, such as site lease contracts, integrated service contracts, etc. The contract classification system is stored in JSON format.

[0114] Optionally, the contract classification system includes:

[0115] {

[0116] "Equipment and Product Procurement Contracts": [...]

[0117] "Lease Agreement": [...]

[0118] "Contracts for Engineering Projects": [...]

[0119] "Technology Contracts": [...]

[0120] "Marketing and promotional contracts": [...]

[0121] "Business Cooperation Contracts": [...]

[0122] "Customer Contracts": [...]

[0123] "Intellectual Property Contracts": [...]

[0124] "Equity-related contracts": [...]

[0125] "Financial and Financial Contracts": [...]

[0126] "Comprehensive Management Contracts": [...]

[0127] {

[0128] "Contract Category": "Consulting Service Contract"

[0129] Business Keywords: "Planning services, market research, market consulting and planning, system design consulting"

[0130] }

[0131] Optionally, based on the correlation between the contract data and the judicial interpretation data, a review point knowledge base is generated, including:

[0132] The contracts in the contract data are classified according to contract type to obtain classified contract data corresponding to different types of contracts.

[0133] Determine the keyword database for the categorized contract data;

[0134] Based on the correlation between the judicial interpretation data and the keyword database, the target judicial interpretation is determined;

[0135] The knowledge base for review points is determined based on the aforementioned target judicial interpretation.

[0136] Optionally, given the numerous judicial interpretations in the aforementioned judicial interpretation data, a large model can be used to accurately filter out target judicial interpretations that are strongly correlated with the keyword data, leveraging its powerful semantic understanding, legal knowledge, and reasoning capabilities.

[0137] Optionally, the contract types include: equal / unequal subject contract types and specific contract types.

[0138] In this embodiment of the application, a series of representative keywords are extracted for each contract subcategory, thereby quickly establishing the association between judicial interpretations and contract types based on the relevance of the keywords to the judicial interpretation database.

[0139] The method for generating a review point knowledge base in this application embodiment can determine a contract classification system by classifying contracts in the contract data, and efficiently filter out target judicial interpretations that are strongly related to contract business by determining keywords for different types of contracts based on the relevance of all keywords in the judicial interpretation database and the keyword database; finally, the review points corresponding to all types of contracts are determined based on the target judicial interpretations, thereby determining the review point knowledge base.

[0140] Optionally, the target judicial interpretation is determined based on the relevance between the judicial interpretation data and the keyword database, including:

[0141] Each judicial interpretation in the data is broken down into multiple clause units; each clause unit includes a complete legal rule.

[0142] The multiple clause units are merged to obtain clause segments;

[0143] The relevance between the clause segments and the keyword database is determined using a large model;

[0144] Based on the relevance, target clause segments related to the keyword database are determined;

[0145] Determine the target judicial interpretation corresponding to the target clause segment.

[0146] It should be noted that the original texts of judicial interpretations are usually quite lengthy. To achieve high efficiency and accuracy in determining the relevance of contracts to business transactions, this application's embodiments break down and preprocess the aforementioned judicial interpretations, such as... Figure 2 As shown:

[0147] While breaking down the text, it is necessary to ensure that each text paragraph used for judgment (the clause unit) is a complete and independent clause, so as to avoid judgment errors caused by the intertwining of clause content;

[0148] Furthermore, given that directly determining the contract category for each clause unit individually is time-consuming, a text merging strategy is introduced to intelligently merge the clause units to obtain the clause segments.

[0149] Specifically, the number of characters in each merged paragraph is strictly controlled to not exceed 2,000 characters. While ensuring the semantic integrity of the text, the number of paragraphs that need to be judged is reduced, thereby improving the judgment efficiency.

[0150] like Figure 3 As shown, the relevance between the clause segments and the keyword database is determined through a large model, including:

[0151] A two-layer loop mechanism is activated, with the outer loop iterating through the contract types sequentially and the inner loop processing each of the decomposed judicial clause units one by one.

[0152] For each paragraph of the judicial interpretation, a large model trained by legal professionals is invoked to accurately assess its relevance to the current contract type based on preset judgment criteria.

[0153] Once at least one legal clause unit is found to be relevant to the contract type, the loop is immediately exited to avoid unnecessary calculations and effectively save computational resources.

[0154] When making judicial interpretations and business relevance judgments based on large models, the locally deployed deepseek-r1-32b model is used, and prompts are designed according to contract business requirements.

[0155] For example, Prompts mainly include the following parts:

[0156] (1) Standardized output format:

[0157] The model is required to evaluate the input judicial interpretation paragraph and return the evaluation result in a preset format, as follows:

[0158] Please determine whether the following legal interpretations are relevant to your contractual business. If relevant, please return "Relevant"; otherwise, please return "Irrelevant".

[0159] This design ensures the consistency and parsability of the model output, facilitating subsequent automated processing;

[0160] (2) Basic rules for judging correlation:

[0161] The model should determine whether a judicial interpretation is relevant to contractual business based on the following fundamental criteria:

[0162]

[0163] (3) Special rules for correlation judgment:

[0164] If a judicial interpretation meets one of the following conditions, it is also considered to be related to contractual business:

[0165]

[0166] (4) Relevance judgment exclusion rule:

[0167] The following situations should be excluded from the scope of contractual business relevance:

[0168]

[0169] (5) Special considerations for relevance assessment:

[0170] To ensure the comprehensiveness and accuracy of the judgment results, the model should follow the following principles during the judgment process:

[0171]

[0172]

[0173] Optionally, the review point knowledge base is determined according to the target judicial interpretation, including:

[0174] The target judicial interpretation is broken down into multiple target judicial interpretation clauses;

[0175] The target judicial interpretation clauses are processed by a large model to obtain target review point data corresponding to multiple target judicial interpretation clauses; the target review point data includes: multiple target review points, as well as the identifier, risk point, content, applicable legal information and review legal basis of each target review point;

[0176] Based on the target review point data, construct the review point knowledge base.

[0177] In this embodiment of the application, based on the selected target judicial interpretation clauses, a large model is used again to identify key information such as core legal provisions, rights and obligations, and exception clauses, and transform them into structured review point definitions to generate potential review point suggestions. These suggestions can be reviewed and confirmed by domain experts to ultimately determine the review points used for contract review, so as to achieve complementary advantages between expert experience and large model capabilities, and improve the efficiency and coverage of review point development.

[0178] In this embodiment of the application, to ensure the accuracy and practicality of the review points during the generation of review points based on a large model, the judicial interpretation clauses are used as the smallest extraction unit for text segmentation. The specific segmentation scheme includes:

[0179] Semantic clause decomposition: A dedicated contract-related text decomposition service is invoked to decompose the original text into semantically independent and logically complete clause units based on the clause numbering system (such as "Article 1", "(I)", "1.", etc.), hierarchical structure and writing conventions in the judicial interpretation text. This decomposition method ensures that each unit corresponds to a legal rule that can be judged independently, avoids semantic confusion across clauses, and provides a high-quality input foundation for subsequent correlation analysis.

[0180] Invalid Fragment Filtering: The validity of the split clause units is evaluated, and fragments without substantive legal content (such as isolated numbers, introductory phrases, empty paragraphs, etc.) are identified and removed. Such fragments are usually short and do not have independent judgment value. If they are retained, it will increase the risk of misjudgment and computational overhead of the large model. Through this filtering mechanism, the signal-to-noise ratio of the input data is significantly improved, redundant reasoning is reduced, and the overall performance of the system is optimized.

[0181] In this embodiment of the application, for each preprocessed text segment, a request message containing specific instructions is constructed and sent to the large model server via an HTTP POST request; the prompt information carried in the request explicitly instructs the large model to extract the review points related to the contract based on the provided legal and judicial interpretations, ensuring that the model can accurately understand the task requirements and generate the corresponding output.

[0182] For example, the Prompt is:

[0183] "Please extract the relevant review points for the contract based on the following legal and judicial interpretations."

[0184] Specific content of the judicial interpretation: {Excerpt from the judicial interpretation}

[0185] Model parameter configuration unit: In the request sent to the large model, the locally deployed deepseek-r1-32b model is used, and the model running parameters are configured, including: setting the system role as an expert in the field of contract review, providing prompts containing specific legal and judicial interpretations for the user role, and hyperparameters such as temperature value, top_p value, top_k value, and maximum token count to optimize the model's output quality and relevance. Specific configuration parameters are shown in Table 2.

[0186] Table 2 Large Model Parameter Configuration

[0187]

[0188] The results extraction and post-processing unit extracts the generated review point text content from the response returned by the large model, performs post-processing operations on the extracted text (such as removing specific marker content), and splits the final review point results and their generation reasons to ensure the accuracy and usability of the output data. The main post-processing tasks include:

[0189]

[0190]

[0191] The review point table generation unit organizes and links the extracted review point names, optimized content, applicable legal information, and review legal basis data to generate review point data in tabular form. A data frame structure is created using a data processing library (such as pandas), and the data is saved as a spreadsheet file. This facilitates expert viewing, analysis, and utilization of the review point information, providing strong support for contract review work. The table includes applicable law, review basis, review risk points, and review name, as shown in Table 3.

[0192] Table 3. Example of a review point table

[0193]

[0194] Optionally, an expert review process can be included to ensure the quality and applicability of the review points are verified.

[0195] Based on the review points in the table and the corresponding legal provisions, the experts rigorously screened and evaluated the potential review points generated by the large model. With their profound legal expertise and practical experience, the experts eliminated review points that did not meet actual business needs or had logical flaws, while confirming the review points that had practical production significance and had been developed and verified, and identified them as key elements in contract review.

[0196] This process not only ensures the accuracy and practicality of the review points, but also further optimizes the completeness and effectiveness of the review point set, providing a solid guarantee for the accurate implementation of subsequent contract review work. During the confirmation process, the system compares the semantic similarity between the new review points and the existing knowledge base, marks potential duplicates, and the confirmed review points will be structured and stored in the review point knowledge base for subsequent contract review system calls. This process ensures the efficient, accurate, and sustainable updating of legal knowledge.

[0197] like Figure 4 The present application also provides a method for determining review content, applied to the review point knowledge base generation method described above, including the following steps:

[0198] Step 401: Perform pre-annotation processing on the contract to be reviewed to determine the annotation information of the contract to be reviewed; the pre-annotation processing includes: large language model pre-annotation and keyword pre-annotation; the annotation information includes: keywords and the position information of the keywords;

[0199] Step 402: Determine the target review points in the review point knowledge base that are related to the annotation information;

[0200] Step 403: Determine the content to be reviewed in the contract to be reviewed based on the target review points and the annotation information.

[0201] It should be noted that the annotation workload for training the key information extraction model in contract review is huge, especially when there are many review labels, resulting in low efficiency. Pre-annotation of key contract information is the core link to improve annotation efficiency, and this application solves this problem through pre-annotation.

[0202] Specifically, a multi-strategy fusion approach is adopted, mainly including: document content parsing and preprocessing, large model semantic pre-annotation, and precise keyword pre-annotation.

[0203] In this embodiment, the original contract document is first standardized and segmented. Then, semantic-level pre-annotation is performed using a Large Language Model (LLM), supplemented by precise keyword matching, to generate pre-annotated results with accurate location information. These results are directly fed back to human annotation platforms such as Docano, significantly reducing the difficulty and time of manual verification and significantly improving annotation efficiency and data quality.

[0204] The method for determining the review content in this application embodiment pre-annotates the contract to be reviewed using large language model pre-annotation and keyword pre-annotation to determine annotation information including the keywords and keyword location information of the contract to be reviewed. Then, it selects the target review point corresponding to the annotation information from the review point knowledge base. Finally, based on the target review point and the annotation information, it determines the content to be reviewed in the contract to be reviewed. This application's solution, through pre-annotation of the contract to be reviewed combined with the generation of the review point knowledge base, achieves automatic extraction of the review content of the contract to be reviewed, improving the efficiency and accuracy of contract review.

[0205] Optionally, the contract to be reviewed undergoes pre-annotation processing to determine the annotation information of the contract to be reviewed, including:

[0206] The contract to be reviewed is split into multiple text segments according to logical units;

[0207] Multiple text segments were pre-annotated using a large model to obtain multiple primary keywords;

[0208] The contract to be reviewed is pre-labeled with keywords to determine the second keyword and the location information of the keyword in the contract to be reviewed; the keyword includes the first keyword and the second keyword.

[0209] The method for determining the content to be examined in this application adopts intelligent segmentation or block processing, which better addresses the variability in the length and information density of contract text, and significantly improves the performance and accuracy of pre-annotation.

[0210] It should be noted that pre-annotating key information at the paragraph level performs significantly better than extracting the entire contract, primarily due to the following considerations:

[0211] (1) Improve extraction robustness: Given the potential decrease in robustness of current large language models (such as DeepSeek-R1 32B) when dealing with extremely long contexts and the accuracy challenge when extracting multiple entities at the same time, decomposing the task into smaller text segments can enable the model to focus more on local information, thereby improving the certainty and accuracy of each extraction.

[0212] (2) Avoiding model capability limitations: Although some models have a long context window, there may still be performance bottlenecks or information loss risks when processing full-text contracts of extreme length (such as tens of thousands of words). Segmented processing can effectively avoid this problem.

[0213] (3) Optimize the output structure: In order to ensure the determinism of each model output and the simplicity of subsequent processing, this scheme is designed to extract one or a group of specific information at each paragraph level, thereby avoiding the result confusion or difficulty in parsing caused by extracting too much information at once;

[0214] (4) Comprehensive capture of multiple instances: Some key information in the contract (such as "company name") may be scattered in different paragraphs in multiple forms (such as full name, abbreviation, and reference in different contexts). By segmenting the process, the system can comprehensively traverse and capture all valid instances, rather than being limited to the first or main mention in the entire text, which is crucial for comprehensive information extraction.

[0215] For example, the contract to be reviewed is split into parts, including:

[0216] Intelligent segmentation and chunking: The pre-processed contract text to be reviewed is intelligently segmented or chunked. This is achieved by calling the project's existing text segmentation service to initially divide long texts into logical units (such as clauses and paragraphs). For a very small number of text segments that still exceed the recommendation context window of the large language model after segmentation, further fine-grained segmentation will be performed to ensure that the size of each text segment meets the model input requirements.

[0217] Large-scale model inference: The segmented text segments are taken as input and sent to a pre-trained large language model via a prompt. The prompt contains a clear role setting (e.g., "You are an expert in the field of contract information extraction"), detailed definitions of the tags to be extracted, and their precise descriptions. The model runs under a temperature parameter set to 0.0 to ensure the determinism and stability of the extraction results, and returns the extracted key information in a unified JSON format, remaining as faithful as possible to the original text of the terms without performing any inference or processing.

[0218] Optionally, keyword pre-tags are performed on the contract to be reviewed to determine the second keyword and the position information of the keyword in the contract to be reviewed, including:

[0219] A string search algorithm is used to pre-annotate keywords in the contract to be reviewed based on a keyword database to obtain a second keyword and the location information of the keyword.

[0220] The keyword database includes keyword data corresponding to various types of contracts; the keyword location information includes the start and end positions of the keyword in the contract to be reviewed.

[0221] Given the potential for inaccurate positioning during short field pre-annotation, this application employs a large model for secondary verification and index recovery. Specifically:

[0222] When multiple instances of pre-annotated short fields exist in the text, causing difficulties in index recovery, the system will trigger the large model to intervene.

[0223] The large model will receive the preliminary results of the pre-annotation and the corresponding text context. By asking the large model questions (e.g., "Please confirm whether the '[pre-annotated content]' I circled is the '[review point name]' in this paragraph?"), the model will utilize its powerful semantic understanding and reasoning capabilities to perform secondary verification of the pre-annotation results.

[0224] The large model can accurately determine whether the boundaries and content of short fields meet the review point requirements based on the context, and assists in accurately locating target information when necessary. This significantly improves the accuracy, robustness, and adaptability to complex scenarios of short field extraction, ensuring the quality of the final review point data.

[0225] For the actual extraction of review points in contract texts, this application adopts a collaborative multi-granularity information extraction strategy to adapt to the characteristics of different types of review points:

[0226] Long-field text classification: For long-field review points involving judgments on the macro-semantic meaning or paragraph categories of text (e.g., breach of contract clauses, confidentiality clauses, etc.), this invention employs a text classification model for processing. This model can classify long text paragraphs, providing contextual clues for subsequent fine-grained extraction or directly outputting classification results;

[0227] Uniform Information Extraction (UIE) for Short Fields: For short field review points that require precise identification and location from contract text (e.g., specific numerical values, dates, names, particular phrases, or word-level legal terms), this invention employs a Uniform Information Extraction (UIE) model. This model efficiently extracts these precise entities or phrases from the text, achieving accurate location of key information.

[0228] Optionally, the contract to be reviewed is split into multiple text segments according to logical units, including:

[0229] Based on the structural features and overall meaning of the contract to be reviewed, the contract to be reviewed is segmented to obtain the first text segment;

[0230] The first text segment is subjected to information extraction processing to obtain the plurality of text segments.

[0231] For long fields in the contract under review that involve macro-level semantics, paragraph classification, or determination of the existence of specific clauses, this application uses a text classification model for segmentation. It does not usually require the extraction of precise phrases or values, but rather the determination of the overall attributes or affiliation of a logical unit, such as identifying the type of large paragraphs like "breach of contract clauses," "confidentiality clauses," or "dispute resolution clauses," or determining whether the contract contains a certain type of agreement.

[0232] Classification models based on deep learning architectures (such as Transformer-based classifiers) are trained on a large number of labeled long text paragraphs, enabling them to learn and recognize the inherent semantic patterns of different types of text; their advantages are:

[0233] Macro-level semantic understanding: It can effectively capture the overall meaning and textual structure features of long texts;

[0234] Structured approach: Provides important contextual clues and scope limits for subsequent finer-grained information extraction; for example, when a paragraph is classified as "payment terms", the short field extraction model can more focused on finding information such as payment amount, method, and date within that paragraph, avoiding searching in irrelevant content, thereby greatly reducing misjudgments and improving efficiency;

[0235] Direct classification output: For certain review points, the classification result itself is the result. For example, when judging the "nature" of a contract (sale, service, lease) or "whether it contains a certain mandatory clause", the text classification model can directly output the final result.

[0236] For short-field review points that require precise identification, location and extraction of specific content from contract text, this application adopts the Universal Information Extraction (UIE) model for processing. These review points usually correspond to key entities or attribute values ​​in the contract, such as "contract name", "contract amount", "contract effective date", "delivery date", etc. The extraction result is required to be an exact string in the text, with its start and end index in the original text.

[0237] The UIE model is a general paradigm that can perform various information extraction tasks such as entity recognition, relation extraction, and event extraction.

[0238] The UIE model used in this application has been trained and fine-tuned with large-scale contract data, enabling it to efficiently extract these precise entities or phrases from text.

[0239] Through this collaborative strategy of multi-granularity models, the present invention can make full use of the advantages of different models to achieve comprehensive, accurate and efficient extraction of various types of review points in contract texts, which is significantly better than single model or coarse-grained extraction methods.

[0240] The method for determining the review content in this application introduces a keyword pre-annotation mechanism as an effective supplement to the semantic understanding of large models. By maintaining a preset keyword dictionary for contract review points, and using an efficient string search algorithm, all keywords and their location information are accurately located and extracted from the full text of the contract. The keyword pre-annotation method makes up for the small number of omissions that may exist in large models (LLM), and is especially suitable for the accurate capture of specific terms, fixed phrases or formatted content, further improving the recall and accuracy of pre-annotation.

[0241] Optionally, the method further includes:

[0242] The pre-annotation results obtained from the large language model pre-annotation and the keyword pre-annotation are fused to obtain the annotation information.

[0243] In the method of this application embodiment, the keywords are accurately recorded in the annotation information, along with the positional information of the keywords (e.g., the start and end character indices of the keywords in the original text). This pre-annotation result with positional information forms the basis for efficient manual verification and can be directly mapped to the text view of annotation platforms such as Doccano.

[0244] Given the high accuracy requirements of contract review, a manual verification step can be retained to ensure the high quality and legal reliability of the output data. This design conforms to the typical human-in-the-loop paradigm of large-model assisted annotation, and is significantly superior to purely manual annotation while ensuring quality.

[0245] For example, by importing the annotation information into professional annotation platforms such as Docano, annotators do not need to start from scratch. Instead, they can quickly review, correct, or supplement the pre-annotated information. This workflow of "pre-annotation + manual verification" significantly shortens the annotation cycle.

[0246] Given that short fields (such as numbers, dates, and abbreviations) frequently appear multiple times in contract texts, and that the initial results of large model pre-annotation only provide values ​​and not precise locations, relying solely on string search methods may lead to inaccurate index positioning or multiple candidate matches. To address this challenge, this application introduces a secondary verification mechanism based on a large model to achieve accurate index recovery and calibration for short fields.

[0247] Optionally, based on the target review points and the annotation information, the content to be reviewed in the contract to be reviewed is determined, including:

[0248] When multiple keywords exist, semantic understanding and reasoning are performed on the keywords to determine the content to be reviewed between the target start position and the target end position.

[0249] In this embodiment of the application, during the extraction of contract information, certain keywords (such as "quantity", "date", "amount") may appear multiple times in the text as short fields, and their literal values ​​are the same.

[0250] For example, the contract may mention "100" in multiple places, one as "quantity" and another as "page number" or "part of the contract number." The initial large model pre-annotation stage usually returns the extracted values, but does not directly provide the precise location of the value in the original text. At this point, if only simple string lookups are relied upon, all matching index positions will be returned, which will create ambiguity.

[0251] For example, the contract text to be reviewed contains the statement: "This contract consists of 100 pages and is made in duplicate. Article 1: Party A purchases 100 units of equipment from Party B at a unit price of RMB 50,000. Contract No.: CG2025-100." Here, "100" appears three times throughout the text, representing "page number," "quantity," and "contract number." The large model may correctly identify "quantity: 100" during the pre-annotation stage, but its output is a structured value and does not include location information. If the system directly locates "100" through string matching, it will obtain multiple candidate positions, making it impossible to determine which corresponds to the "quantity" field.

[0252] In this embodiment of the application, when using the string matching method to find a short field value (e.g., "100" in "Quantity: 100") identified by the initial large model pre-annotation, if the value has multiple matches, that is, the length of the returned index list is greater than 1, the system will trigger the large model secondary confirmation process; this ensures that the large model is only introduced when needed, thus optimizing the use of computing resources.

[0253] Furthermore, for each ambiguous candidate index position, a probing input with contextual information can be constructed and sent to the large model. Its powerful semantic understanding and reasoning capabilities can be used for secondary verification to accurately determine whether the short field at that position truly represents the specified review point.

[0254] For example, in order for a large model to make accurate judgments, the input needs to be carefully constructed, specifically including:

[0255] (1) Context extraction and highlighting: Extract context fragments containing candidate short fields from the original text segments and highlight the short fields using specific tags (such as [* / ^ / * content * / ^ / *]). This operation clearly indicates the model's focus, enabling it to focus on the tagged content and its surrounding context.

[0256] (2) Introduction of review point definition: Combined with the detailed definition of the review point to which the short field belongs, the model is provided with the basis for judgment and domain knowledge to help it understand the semantic boundary and features of the information to be confirmed;

[0257] (3) Question formatting: Combine the above information into a clear judgment question to guide the model to make a binary classification (yes / no) decision; only when the large model clearly returns a positive judgment result (such as "yes") will the candidate index position be confirmed as a valid instance of the review point and included in the final pre-labeling result.

[0258] The secondary confirmation mechanism in this application significantly improves the accuracy, robustness, and adaptability to complex scenarios of short field extraction, especially when dealing with easily confused numbers, dates, or abbreviations, ensuring the quality of the final review point data.

[0259] For example, the prompts used are carefully designed to ensure that the large model can efficiently and accurately complete the secondary confirmation task for short fields; the prompts mainly consist of the following parts:

[0260]

[0261] Through the design of the aforementioned prompts, the large model can leverage its powerful semantic understanding and reasoning capabilities to make an accurate semantic judgment on whether a short field at a given location truly represents a specified review point, based on the context of the highlighted field and the definition of the review point. Only when the large model explicitly returns a positive judgment result (such as "yes") will the candidate index location be confirmed as a valid instance of the review point and included in the final pre-labeled results. This secondary confirmation mechanism significantly improves the accuracy, robustness, and adaptability to complex scenarios of short field extraction, especially when dealing with easily confused numbers, dates, or abbreviations, ensuring the quality of the final review point data.

[0262] Compared with existing prompt designs, the prompts in this application are specifically optimized for the task of "semantic disambiguation of short fields with multiple candidates". By guiding the model's attention through highlighting, providing judgment criteria through review point definition, and ensuring controllable output through standardized instructions, the three aspects work together to significantly improve the judgment accuracy in high-noise text environments.

[0263] Through the aforementioned prompts, the large model can leverage its powerful semantic understanding and reasoning capabilities to make a precise semantic judgment on whether a short field at a given location truly represents a specified review point, based on the context of the highlighted field and the definition of the review point. Only when the large model explicitly returns a positive result (such as "yes") will the candidate index location be confirmed as a valid instance of the review point and included in the final pre-labeled results. This secondary confirmation mechanism significantly improves the accuracy, robustness, and adaptability to complex scenarios of short field extraction, especially when dealing with easily confused numbers, dates, or abbreviations, ensuring the quality of the final review point data.

[0264] The prompts in this application are specifically optimized for the task of "semantic disambiguation of short candidate fields". They guide the model's attention through highlighting, provide judgment criteria through review point definitions, and ensure controllable output through standardized instructions. These three elements work synergistically to significantly improve judgment accuracy in noisy text environments. Experiments show that in secondary confirmation tasks involving fields such as "quantity" and "amount", this prompt design improves accuracy to 96.5%, a 12.8% improvement over the baseline version without definitions and highlighting.

[0265] To achieve comprehensive and accurate extraction of various review points in contract texts, this application adopts a multi-granularity information extraction model collaborative working strategy. This strategy selects the most suitable model for processing based on the characteristics of the review points (macro-semantic judgment or fine-grained entity recognition), and through the collaborative cooperation between models, significantly improves the overall accuracy and robustness of the extraction.

[0266] Building upon this foundation, this application further constructs a dedicated information extraction architecture for contract review scenarios, comprising a text classification model for long-field review points and a unified information extraction (UIE) model for short-field review points. The text classification model identifies the overall semantic category of clauses such as "liability for breach of contract" and "dispute resolution," while the UIE model accurately extracts key entities such as "amount" and "date" and their locations from specified paragraphs. The two models are not simply strung together sequentially, but rather achieve efficient linkage through a collaborative mechanism that "limits the extraction scope based on classification results": when a paragraph is classified as "payment terms," ​​the system only initiates extraction of related short fields such as "payment amount" and "payment method" within that paragraph, avoiding invalid searches in irrelevant content. This design significantly reduces the false positive rate of short-field extraction and improves overall processing efficiency. Experimental results show that on typical contract samples, this collaborative mechanism improves the accuracy of short-field extraction by approximately 18% while reducing redundant computation by approximately 40%.

[0267] For long-field review points in contract review involving macro-level semantics, paragraph category judgment, or determination of the existence of specific clauses, this application adopts a text classification model for processing, specifically:

[0268] The text classification model described above is based on a deep learning architecture (such as a Transformer-based classifier). By training on a large number of labeled long text paragraphs, it learns and recognizes the inherent semantic patterns of different text types. Its advantages are:

[0269] Macro-level semantic understanding: It can effectively capture the overall meaning and textual structure features of long texts;

[0270] Structured approach: Provides important contextual clues and scope constraints for subsequent finer-grained information extraction. For example, when a paragraph is categorized as "payment terms," ​​the short-field extraction model can more effectively search for information such as payment amount, method, and date within that paragraph, avoiding searches in irrelevant content, thereby significantly reducing false positives and improving efficiency.

[0271] Direct classification output: For certain review points, the classification result itself is the result. For example, when judging the "nature" of a contract (sale, service, lease) or "whether it contains a certain mandatory clause", the text classification model can directly output the final result.

[0272] For short-field review points that require precise identification, location and extraction of specific content from contract text, this application adopts the Uniform Information Extraction (UIE) model for processing.

[0273] The UIE model is a general paradigm capable of performing various information extraction tasks such as entity recognition, relation extraction, and event extraction. The UIE model used in this solution has been trained and fine-tuned with large-scale contract data, enabling it to efficiently extract precise entities or phrases from text. Its core advantages are:

[0274] Precise positioning: It can accurately identify the boundaries (start and end positions) of short fields in the text, providing high-quality annotation-level information.

[0275] Unified multi-task: It handles the extraction of multiple types of short fields within a single model framework, simplifying the complexity of model management and maintenance.

[0276] High robustness: It can cope with the diversity of expressions, abbreviations, typos and other situations that may exist in the text, and ensure the stability and accuracy of the extraction results.

[0277] This invention, through intelligent orchestration of the text classification model and the unified information extraction model, forms a highly efficient multi-granularity information extraction pipeline, rather than simply operating them independently. Specifically:

[0278] Pre-structuring: The long-field text classification model first performs preliminary structuring and semantic classification of the contract text. Its output classification results provide important macro-context and semantic guidance for the subsequent short-field UIE model.

[0279] Targeted Extraction: Based on the long-field classification results, the short-field UIE model can be applied to specific types of paragraphs or clauses, thereby narrowing its search scope and improving extraction efficiency and accuracy. For example, the UIE model will only extract relevant information such as "payment method" or "prepayment ratio" from paragraphs identified as "payment terms".

[0280] Cross-validation and reinforcement: The results of the two models can be cross-validated and complemented. Long-field classification can provide corroboration for short-field classification, and conversely, the absence or presence of certain short fields can infer the accuracy of long-field classification.

[0281] Through this collaborative strategy of multi-granularity models, the present invention can make full use of the advantages of different models to achieve comprehensive, accurate and efficient extraction of various types of review points in contract texts, which is superior to single model or coarse-grained extraction methods.

[0282] like Figure 5 As shown, this application uses a large language model to determine the relevance of judicial interpretations to contracts from a vast amount of data. The system leverages the semantic understanding, legal knowledge, and reasoning capabilities of the large model to assess the degree of correlation between various judicial interpretations and contractual transactions, efficiently filtering out judicial interpretations that are strongly relevant to contractual transactions.

[0283] Building upon this foundation, this application utilizes a large-scale model to automatically generate potential review point suggestions from these selected judicial interpretations. The large-scale model can identify core legal provisions, rights and obligations, and exception clauses within the judicial interpretations, transforming them into structured review point definitions. Subsequently, domain experts review and confirm the review point list generated by the large-scale model, selecting the review points ultimately needed for contract review. This achieves an organic combination of expert experience and the large-scale model's generation capabilities, significantly improving the efficiency and breadth of review point development.

[0284] Meanwhile, to significantly reduce the cost and workload of traditional manual annotation, this application utilizes large-scale models for pre-annotation. For review point data requiring model training, the large-scale model, with its powerful language understanding and generation capabilities, can perform preliminary information extraction and classification (i.e., pre-annotation) on a large amount of unannotated contract text, providing a high-quality starting point for manual annotation. Subsequently, human experts only need to efficiently verify and correct the pre-annotation results of the large-scale model, rather than annotating from scratch, thereby greatly improving annotation efficiency and reducing data costs. This "large-scale model pre-annotation + manual verification" model effectively replaces the inefficient process of traditional purely manual annotation.

[0285] The various methods of the embodiments of this application have been described above. Apparatus for implementing the above methods will now be provided.

[0286] like Figure 6 As shown in the figure, this application embodiment also provides a knowledge base generation device 600, including:

[0287] The first determining module 601 is used to determine judicial interpretation data based on a legal database; the judicial interpretation data includes multiple original texts of judicial interpretations.

[0288] The second determining module 602 is used to determine contract data according to industry standards; the contract data includes various types of contracts.

[0289] The generation module 603 is used to generate a review point knowledge base based on the correlation between the contract data and the judicial interpretation data; the review point knowledge base includes multiple review points for reviewing the contract; the review points are the review points in the judicial interpretation data that correspond to the judicial interpretations related to the contract data.

[0290] The examination point knowledge base generation device provided in this application embodiment determines judicial interpretation data through a legal database and contracts including various types of contracts according to industry standards; then, based on the correlation between the contract data and the judicial interpretation data, it generates an examination point knowledge base, providing data foundation support for the subsequent extraction of examination points in the contracts to be examined.

[0291] like Figure 7 As shown in the illustration, this application also provides an examination content determination device 700, comprising:

[0292] The third determining module 701 is used to perform pre-annotation processing on the contract to be reviewed, and to determine the annotation information of the contract to be reviewed; the pre-annotation processing includes: large language model pre-annotation and keyword pre-annotation; the annotation information includes: keywords and the position information of the keywords;

[0293] The fourth determining module 702 is used to determine the target review points in the review point knowledge base that are related to the annotation information;

[0294] The fifth determining module 703 is used to determine the content to be reviewed in the contract to be reviewed based on the target review points and the annotation information.

[0295] The review content determination device in this application embodiment pre-annotates the contract to be reviewed through large language model pre-annotation and keyword pre-annotation, determining annotation information including keywords and keyword location information of the contract to be reviewed; thereby selecting the target review point corresponding to the annotation information from the review point knowledge base, and finally determining the content to be reviewed in the contract to be reviewed based on the target review point and the annotation information. This application's solution, through pre-annotation of the contract to be reviewed combined with the generation of the review point knowledge base, achieves automatic extraction of the review content of the contract to be reviewed, improving the efficiency and accuracy of contract review.

[0296] Another embodiment of the network device in this application, such as Figure 8 As shown, it includes a transceiver 810, a processor 800, a memory 820, and a program or instructions stored in the memory 820 and executable on the processor 800; when the processor 800 executes the program or instructions, it implements the various processes of the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0297] The transceiver 810 is used to receive and send data under the control of the processor 800.

[0298] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 800) and memory (memory 820). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 810 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 800 is responsible for managing the bus architecture and general processing, and the memory 820 can store data used by the processor 800 during operation.

[0299] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0300] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described examination point confirmation method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0301] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0302] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0303] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for generating a review point knowledge base, characterized in that, include: Based on legal databases, determine the judicial interpretation data; The judicial interpretation data includes the original texts of multiple judicial interpretations; Determine contract data according to industry standards; The contract data includes various different types of contracts; A review point knowledge base is generated based on the correlation between the contract data and the judicial interpretation data; the review point knowledge base includes multiple review points for reviewing the contract; the review points are the review points corresponding to the judicial interpretations related to the contract data in the judicial interpretation data.

2. The method according to claim 1, characterized in that, Based on the correlation between the contract data and the judicial interpretation data, a review point knowledge base is generated, including: The contracts in the contract data are classified according to contract type to obtain classified contract data corresponding to different types of contracts. Determine the keyword database for the categorized contract data; Based on the correlation between the judicial interpretation data and the keyword database, the target judicial interpretation is determined; The knowledge base for review points is determined based on the aforementioned target judicial interpretation.

3. The method according to claim 2, characterized in that, Based on the correlation between the judicial interpretation data and the keyword database, the target judicial interpretation is determined, including: Each judicial interpretation in the data is broken down into multiple clause units; each clause unit includes a complete legal rule. The multiple clause units are merged to obtain clause segments; The relevance between the clause segments and the keyword database is determined using a large model; Based on the relevance, target clause segments related to the keyword database are determined; Determine the target judicial interpretation corresponding to the target clause segment.

4. The method according to claim 2, characterized in that, Based on the aforementioned target judicial interpretation, the knowledge base for review points is determined, including: The target judicial interpretation is broken down into multiple target judicial interpretation clauses; The target judicial interpretation clauses are processed by a large model to obtain target review point data corresponding to multiple target judicial interpretation clauses; the target review point data includes: multiple target review points, as well as the identifier, risk point, content, applicable legal information and review legal basis of each target review point; Based on the target review point data, construct the review point knowledge base.

5. A method for determining review content, applied to the review point knowledge base generation method as described in any one of claims 1 to 4, characterized in that, include: The contract to be reviewed is pre-annotated to determine the annotation information of the contract to be reviewed; The pre-annotation process includes: large language model pre-annotation and keyword pre-annotation; the annotation information includes: keywords and the location information of the keywords; Identify the target review points in the review point knowledge base that are related to the annotation information; Based on the target review points and the annotation information, the content to be reviewed in the contract to be reviewed is determined.

6. The method according to claim 5, characterized in that, The contract to be reviewed is pre-annotated to determine the annotation information of the contract to be reviewed, including: The contract to be reviewed is split into multiple text segments according to logical units; Multiple text segments were pre-annotated using a large model to obtain multiple primary keywords; The contract to be reviewed is pre-labeled with keywords to determine the second keyword and the location information of the keyword in the contract to be reviewed; the keyword includes the first keyword and the second keyword.

7. The method according to claim 6, characterized in that, The contract to be reviewed is pre-annotated with keywords to determine the second keyword and the position information of the keyword in the contract to be reviewed, including: A string search algorithm is used to pre-annotate keywords in the contract to be reviewed based on a keyword database to obtain a second keyword and the location information of the keyword. The keyword database includes keyword data corresponding to various types of contracts; the keyword location information includes the start and end positions of the keyword in the contract to be reviewed.

8. The method according to claim 6, characterized in that, The contract to be reviewed is split into multiple text segments according to logical units, including: Based on the structural features and overall meaning of the contract to be reviewed, the contract to be reviewed is segmented to obtain the first text segment; The first text segment is subjected to information extraction processing to obtain the plurality of text segments.

9. The method according to claim 8, characterized in that, Based on the target review points and the annotation information, the content to be reviewed in the contract to be reviewed is determined, including: When multiple keywords exist, semantic understanding and reasoning are performed on the keywords to determine the content to be reviewed between the target start position and the target end position.

10. A knowledge base generation device, characterized in that, include: The first determination module is used to determine judicial interpretation data based on the legal database; The judicial interpretation data includes the original texts of multiple judicial interpretations; The second determining module is used to determine contract data according to industry standards; the contract data includes various types of contracts. The generation module is used to generate a review point knowledge base based on the correlation between the contract data and the judicial interpretation data; the review point knowledge base includes multiple review points for reviewing the contract; the review points are the review points corresponding to the judicial interpretations related to the contract data in the judicial interpretation data.

11. A device for determining the content to be reviewed, characterized in that, include: The third determining module is used to pre-annotate the contract to be reviewed and determine the annotation information of the contract to be reviewed; The pre-annotation process includes: large language model pre-annotation and keyword pre-annotation; the annotation information includes: keywords and the location information of the keywords; The fourth determining module is used to determine the target review points in the review point knowledge base that are related to the annotation information; The fifth determining module is used to determine the content to be reviewed in the contract to be reviewed based on the target review points and the annotation information.

12. A network device, characterized in that, include: Transceiver, processor, memory, and programs or instructions stored in the memory and executable on the processor; When the processor executes the program or instructions, it implements the steps of the method as described in any one of claims 1 to 4; or, when the processor executes the program or instructions, it implements the steps of the method as described in any one of claims 5 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 4; or, when executed by a processor, the computer program implements the steps of the method as described in any one of claims 5 to 9.

14. A computer program product, characterized in that, The method includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 4; or, when executed by a processor, the computer instructions implement the steps of the method as described in any one of claims 5 to 9.

Images