System clause auditing method, device and equipment, storage medium and product

Abstract

The invention discloses a system clause auditing method and device, equipment, a storage medium and a product, and relates to the technical field of clause auditing, the method comprises the following steps: pairing superior system file clauses and subordinate system file clauses obtained through pre-analysis to obtain superior and subordinate clause pairs; performing quantitative correlation filtering on the upper-level and lower-level clause pairs through a pre-trained quantitative correlation identification model to obtain upper-level and lower-level quantitative correlation clause pairs; inputting the upper and lower level quantitative related clause pairs into a pre-trained upper and lower level system comparison large model to obtain a system auditing clue, quickly eliminating non-quantitative irrelevant clause pairs through a pre-trained quantitative correlation model, and then inputting the quantitative and related clause pairs into the upper and lower level system comparison large model to perform automatic auditing. The auditing clues between the upper-level and lower-level system terms are output in batches, so that the accuracy and efficiency of auditing the system terms are improved, and the auditing cost is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of clause auditing, and in particular to a method, device, equipment, storage medium and product for auditing system clauses. Background Art

[0002] As an effective internal supervision mechanism, system audit can help enterprises evaluate the quality of their organizational management system and ensure that the organizational management system of group enterprises complies with national laws, implements national guidelines and policies, and complies with the business objectives and decisions of enterprises. When the state issues some system requirements, each group and subsidiary should also compile the system in accordance with the relevant requirements. When designing the content of the lower-level system, it should follow the requirements of the upper-level system and make relevant requirements for the terms of the system to ensure the comprehensiveness, effectiveness and unity of the system.

[0003] At present, the method for judging whether the system clauses of digital content in the superior and subordinate systems meet the audit requirements is mainly to parse the basis system in the massive system document library, establish the knowledge graph of the current system, and then find its superior and subordinate systems based on the system knowledge graph, and parse the clauses of the system documents, and then find the relevant clauses through ES Cartesian product matching, and output the audit clues according to the established quantitative rules. After the audit clues are output, the complete clue content is output again after confirmation by the auditor.

[0004] The above method of matching upper and lower level system documents based on knowledge graph, parsing system documents into clauses and then performing ES Cartesian product clause pairing is often inefficient due to the large number of systems and the use of Cartesian product comparison for all of them. The audit clues between clause pairs derived based on quantitative rules will be omitted due to the continuous updating of rules, and the audit clues derived are only audit points without specific reasons for the clues. The output audit points also require professional audit teachers to supplement detailed clues, which requires high labor costs and a long time. Therefore, the existing system audit technical solutions have problems such as low coverage, low accuracy, long audit process, low audit efficiency and high cost.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a system clause audit method, device, equipment, storage medium and computer program product, aiming to improve the accuracy and efficiency of system clause audit and reduce audit costs.

[0007] To achieve the above purpose, this application proposes a system clause audit method, which includes:

[0008] Pair the clauses of the superior system document and the clauses of the subordinate system document obtained in advance, and obtain the superior-subordinate clause pair;

[0009] Performing quantitative correlation filtering on the upper-lower clause pairs through a pre-trained quantitative correlation identification model to obtain upper-lower clause pairs;

[0010] The pairs of quantitatively related clauses between superiors and subordinates are input into a pre-trained large model for comparing superior-subordinate systems to obtain system audit clues.

[0011] In one embodiment, the training process of the quantitative correlation identification model includes:

[0012] Obtain superior-subordinate training clause pairs and pre-built quantized rule expressions;

[0013] According to the quantization rule expression, the upper and lower training clause pairs are quantized and labeled with quantization terms to obtain upper and lower quantized related training clause pairs;

[0014] Iteratively training the pre-built quantization correlation identification initial model based on the upper and lower level quantization correlation training clauses, and obtaining clause correlation results;

[0015] If the relevance identification accuracy corresponding to the clause relevance result meets the preset accuracy threshold, the quantitative relevance identification model is obtained.

[0016] In one embodiment, the upper-lower level training clause pair includes an upper-level clause sentence and a lower-level clause sentence, the quantization rule expression includes a word-removed quantization rule expression and a related rule expression, and the step of performing quantization relevance labeling and quantization word labeling on the upper-lower level training clause pair according to the quantization rule expression to obtain the upper-lower level quantization related training clause pair includes:

[0017] Traversing the upper and lower training clause pairs, and concatenating the upper clause sentences and the corresponding lower clause sentences into upper and lower clause sentences;

[0018] According to the word removal quantification rule expression, fixed keywords of the upper and lower clause sentences are removed to obtain the upper and lower clause clean sentences;

[0019] Determine whether the upper and lower clause clean statements are related clause statements according to the related rule expressions;

[0020] In response to the upper and lower clause clean sentences being related clause sentences, marking the upper and lower training clause pairs corresponding to the upper and lower clause clean sentences as upper and lower related training clause pairs;

[0021] According to the relevant regular expressions, the target quantification words in the upper-lower level related training clause pairs are matched and marked to obtain the upper-lower level quantification related training clause pairs.

[0022] In one embodiment, the step of iteratively training the pre-built quantization correlation identification initial model based on the upper and lower level quantization correlation training clauses and obtaining the clause correlation results includes:

[0023] Perform sentence embedding and pooling processing on the upper and lower level quantization related training clause pairs to obtain the upper level quantization clause sentence embedding vector and the lower level quantization clause sentence embedding vector;

[0024] Calculate the difference vector of the upper and lower quantified clause sentences based on the upper and lower quantified clause sentence embedding vectors;

[0025] Based on the difference vector of the upper and lower level quantitative clause statements, probability distribution conversion is performed to obtain the clause relevance result.

[0026] In one embodiment, if the accuracy of the relevance identification corresponding to the clause relevance result meets a preset accuracy threshold, the step of obtaining the quantitative relevance identification model further includes:

[0027] Obtain the relevance results of the terms;

[0028] Counting the model recognition correct data and model recognition incorrect data corresponding to the clause relevance results, wherein the model recognition correct data includes relevant recognition correct data and irrelevant recognition correct data, and the model recognition incorrect data includes relevant recognition incorrect data and irrelevant recognition incorrect data;

[0029] Calculate the model accuracy based on the relevant identified correct data and the irrelevant identified incorrect data;

[0030] Calculating a model recall rate based on the relevant correctly identified data and the relevant incorrectly identified data;

[0031] The relevance identification accuracy corresponding to the clause relevance result is calculated based on the model accuracy and model recall.

[0032] In one embodiment, the training process of the upper-lower level system comparison large model includes:

[0033] Obtaining a superior-subordinate quantitative clause audit clue data set and a pre-set business scenario prompt, wherein the superior-subordinate quantitative clause audit clue data set includes superior-subordinate quantitative clauses and corresponding superior-subordinate quantitative clause audit clues;

[0034] Generate a large model training input sentence based on the superior-subordinate quantitative terms and business scenario prompts;

[0035] Based on the large model training input sentence, a pre-built initial large model based on system comparison is trained to obtain a large model based on system comparison training;

[0036] Repeat data training and parameter optimization according to the large model based on system comparison training, and obtain the training model output result;

[0037] If the degree of fit between the training output result and the audit clues of the superior-subordinate quantitative clauses is greater than a preset fit threshold, the superior-subordinate system comparison model is obtained.

[0038] In one embodiment, before the step of performing quantitative correlation filtering on the upper and lower level clause pairs through a pre-trained quantitative correlation identification model to obtain the upper and lower level quantitative correlation clause pairs, the step further includes:

[0039] Generate a text array of upper-lower-level quantified clause pairs based on the upper-lower-level quantified clause pairs;

[0040] Based on the upper and lower quantization clauses, the text array is encoded and converted to obtain normalized upper and lower quantization clause pair sentence embeddings, wherein the normalized upper and lower quantization clause pair sentence embeddings include normalized upper quantization clause sentence embeddings and normalized lower quantization clause sentence embeddings;

[0041] Calculate the sentence embedding dot product between the normalized upper-level quantization clause sentence embedding and the normalized lower-level quantization clause sentence embedding to obtain the sentence embedding dot product result of the upper-level quantization clause pair;

[0042] The step of performing quantitative correlation filtering on the upper and lower level clause pairs through the pre-trained quantitative correlation identification model to obtain the upper and lower level quantitative correlation clause pairs comprises:

[0043] The sentence embedding dot product results of the superior-subordinate clause pairs are subjected to quantitative correlation filtering through a pre-trained quantitative correlation identification model to obtain superior-subordinate quantitative correlation clause pairs.

[0044] In addition, to achieve the above purpose, the present application also proposes a system clause audit device, which includes:

[0045] A matching module is used to match the clauses of the superior system document and the clauses of the subordinate system document obtained in advance, so as to obtain a pair of superior and subordinate clauses;

[0046] A filtering module, used for performing quantitative correlation filtering on the upper and lower clause pairs through a pre-trained quantitative correlation identification model to obtain upper and lower quantitative correlation clause pairs;

[0047] The output module is used to input the quantitative related clause pairs between superiors and subordinates into a pre-trained superior-subordinate system comparison model to obtain system audit clues.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a system clause audit device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the system clause audit method described above.

[0049] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the system clause audit method described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the system clause audit method as described above.

[0051] One or more technical solutions proposed in the present application obtain superior-subordinate clause pairs by pairing the clauses of the superior system document and the subordinate system document obtained by pre-analysis; perform quantitative correlation filtering on the superior-subordinate clause pairs through a pre-trained quantitative correlation identification model to obtain superior-subordinate quantitative correlation clause pairs; input the superior-subordinate quantitative correlation clause pairs into a pre-trained superior-subordinate system comparison large model to obtain system audit clues, and after traversing the superior-subordinate system document clauses and pairing them through the above method, perform clause quantitative correlation filtering through a pre-trained quantitative correlation model to quickly exclude non-quantitative irrelevant clause pairs, and then input the quantified and relevant clause pairs into the superior-subordinate system comparison large model, automatically judge all the screened out quantified and relevant clause pairs, and batch output the audit clues between the superior-subordinate system clauses, thereby significantly improving the accuracy and efficiency of system clause audits and reducing audit costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1A flowchart of a method for auditing system clauses provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of the system architecture of an institutional audit method combining quantitative model screening and large models for this application;

[0056] Figure 3 A flowchart of another system clause audit method provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the automatic annotation process of the quantitative correlation of the upper and lower clauses of a system clause audit method provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the BIO automatic annotation process of quantification words in the upper and lower clauses of a system clause audit method provided in an embodiment of the present application;

[0059] Figure 6 A flowchart of another method for auditing system clauses provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of the module structure of the system clause audit device provided in the embodiment of the present application;

[0061] Figure 8 A schematic diagram of the device structure of the hardware operating environment involved in the system clause audit method provided in the embodiment of the present application.

[0062] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0063] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0064] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0065] The main solution of the embodiment of the present application is: facing the technical problem of giving the output of relevant audit clues at the clause level from the massive system documents, a method for automatically generating audit clues of system documents combining quantitative model screening and large model is proposed. Starting from the massive audit system library data source, the system documents are first matched, and the subordinate system related to the superior is found according to the system knowledge map. Match the superior and subordinate system file pairs, and then parse the system files. Use the PDF2word tool and the article splitting tool to parse the system files into clauses, that is, split them from the level of the system files to the level of the clauses. Then match the clauses, and all the clauses of the superior system and all the clauses of the subordinate system are matched according to the Cartesian product. Due to the full amount of clause matching, a large number of irrelevant and non-quantitative clause pairs will be generated. First, the SentBERT quantitative correlation model is trained. When annotating the data, only the superior and subordinate clause pairs are clause pairs containing quantitative content, and the superior and subordinate clauses belong to the same field and describe the same subject. Only then are they judged as relevant clause pairs, and subsequent audit clues are output, otherwise they are judged as irrelevant and no subsequent processing is performed. Train the SentBERT model to judge all the clause pairs matched by the ES Cartesian product. This step can quickly screen out the clause pairs that need to be output as clues, play a filtering role, and greatly improve efficiency. When there is a rule update, re-label the data and retrain the model for rapid iteration. Finally, based on the manually labeled audit clue data, perform SFT fine-tuning on the large language model to obtain a large model for comparing the superior and subordinate systems. Automatically judge all the quantitative and relevant clause pairs screened out by the correlation model in the previous step, and batch output the audit clues between the superior and subordinate system clauses.

[0066] The present application provides a solution, which obtains pairs of superior and subordinate clauses by traversing and matching the clauses of superior system documents and subordinate system documents obtained based on pre-analysis; performs quantitative correlation filtering on the superior and subordinate clause pairs through a pre-trained quantitative correlation identification model to obtain superior and subordinate quantitative correlation clause pairs; inputs the superior and subordinate quantitative correlation clause pairs into a pre-trained superior and subordinate system comparison large model to output system audit clues, and after traversing and matching the clauses of the superior and subordinate system documents through the above method, performs clause quantitative correlation filtering through a pre-trained quantitative correlation model to quickly exclude non-quantitative irrelevant clause pairs, and then inputs the quantified and relevant clause pairs into the superior and subordinate system comparison large model, automatically judges all the screened quantified and relevant clause pairs, and batch outputs audit clues between the superior and subordinate system clauses, thereby significantly improving the accuracy and efficiency of system clause audits and reducing audit costs.

[0067] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a cloud server, a system clause audit system, etc. The following takes the system clause audit system as an example to illustrate this embodiment and the following embodiments.

[0068] Based on this, the present application embodiment provides a method for auditing system clauses, referring to Figure 1 , Figure 1 A flowchart of a system clause audit method provided for an embodiment of the present application.

[0069] In this embodiment, the system clause audit method includes steps S1000 to S4000:

[0070] Step S1000: Pair the pre-parsed upper and lower system document clauses with each other to obtain a pair of upper and lower clauses;

[0071] It should be noted that the system clause audit method proposed in this embodiment is specifically a method for automatically generating system document audit clues by combining quantitative model screening and large model. Figure 2 , Figure 2 This is a schematic diagram of the system architecture of a system audit method that combines quantitative model screening and a large model for this application. The method mainly consists of five parts: the first part is system document matching, the second part is system document parsing, the third part is clause matching, the fourth part is the SentBERT quantitative correlation model, and the fifth part is the upper and lower system comparison large model. The overall process of the system includes finding subordinate system documents related to superior system documents from a massive system document library based on the system knowledge graph, parsing the system documents into clauses using tools such as PDF2word and article splitting tools, and then matching the clauses of the superior system documents with the clauses of the subordinate system documents according to the Cartesian product, and based on the SentBERT quantitative correlation model, quickly filtering out non-related and non-quantitative clause pairs, retaining only quantitative and relevant clause pairs, and finally outputting audit clues based on the upper and lower system comparison large model.

[0072] In addition, it should be noted that the system audit library system often stores a large number of superior and subordinate system documents, including various legal and regulatory system documents, and system documents formulated by downstream companies. The superior and subordinate system documents in the system are linked through the knowledge graph. You can directly enter the superior system file, call the program, and search for related subordinate system documents. For example, the superior system "XX Bidding and Tendering Management Measures", the retrieved subordinate systems include "XX Company Bidding and Tendering Management Implementation Rules", "XX Company Bidding and Tendering Management Measures (V3.0)", "XX Company Bidding and Tendering Management Measures (2015 Revised Edition)", "XX Company Procurement Management Measures", "XX Company Bidding and Tendering Management Measures (2016 Revised)", "XX Company Bidding and Tendering Management Measures (2018 Edition)", "XX Company Bidding and Tendering Management Measures", "XX Company Bidding and Tendering Management Measures (2016 Edition)", "XX Company Bidding and Tendering Management Implementation Rules", "XX Company Bidding and Tendering Management Measures (2017 Edition)"; one superior system document corresponds to approximately 10 subordinate systems. For these superior and subordinate files, the system file parsing capability in the system audit library can be called to perform system parsing on PDF, word documents, etc. The system files can be parsed into single clauses, and divided according to "first clause", "second clause"... "nth clause". The system clauses are the semantic units with the smallest strength and are stored in the database. A system document can be parsed into approximately 50 clauses. In addition, the system file clause parsing in the embodiments of the present application can also choose manual parsing or traditional parsing methods such as knowledge graph-based parsing or regular expression parsing.

[0073] In addition, it should be noted that the parsed superior and subordinate system clauses can be paired to obtain pairs of system documents with superior and subordinate associations, where each superior clause of each superior system document is paired with each clause of the relevant subordinate system document by Cartesian product, and the pairing follows the following calculation method: X×Y={(X,y)|X∈X,y∈Y}, where X represents the set of superior clauses, X represents a single superior clause, Y represents the set of subordinate clauses, and y represents a single subordinate clause. Through this step of operation, the association of all clauses can be completed. The amount of clause data obtained by this step of pairing is very large, and there will be many non-related paired clause pairs. For example, in a feasible implementation, after parsing a superior document, there are 50 clauses, and 10 related subordinate systems are found. Each subordinate system also parses out 50 clauses. After the clauses are paired, 25,000 pairs of superior and subordinate clauses will be generated.

[0074] Step S2000: Performing quantitative correlation filtering on the upper-lower level clause pairs through a pre-trained quantitative correlation identification model to obtain upper-lower level quantitative correlation clause pairs;

[0075] It should be noted that, in this embodiment, the upper and lower clause pairs generated in the above steps are quantitatively related filtered by a pre-trained quantitative correlation identification model, so as to obtain quantified and related upper and lower clauses, that is, upper and lower quantitative related clause pairs. Specifically, this embodiment is based on the SentBERT quantitative correlation model to quickly filter out non-related and non-quantified clause pairs, and only retain quantified and related clause pairs. The SentBERT model is an optimization model of the BERT model, which is specifically used to generate sentence-level embedding representations. Through fixed-size sentence embedding, optimized network structure, fine-tuning mechanism and efficient similarity comparison, the SentBERT model is more suitable for processing text classification tasks. SentBERT performs well in these tasks. In this embodiment, the SentBERT quantitative correlation model is used to perform sentence-level embedding representations on the input upper and lower clauses. Through its optimized network structure and similarity comparison, and the reasoning optimization of data parallel computing, the subordinate system clauses that are related to the superior system and are quantified are quickly and accurately screened out, and the system documents quickly filter out non-related and non-quantified clause pairs, and only retain the upper and lower system clause pairs that are quantified and related, thereby greatly improving the relevance and speed of clause pairing.

[0076] Step S3000: Input the pair of quantitative related clauses between superiors and subordinates into a pre-trained large model for comparing superior and subordinate systems to obtain system audit clues.

[0077] It should be noted that, in this embodiment, after the SentBERT quantitative correlation model is used to quickly filter out non-relevant and non-quantitative clause pairs and obtain the superior-subordinate quantitative correlation clause pairs, the audit clues are output through the pre-trained superior-subordinate system comparison model, wherein the superior-subordinate system comparison model is based on the superior-subordinate system comparison model, that is, the superior-subordinate system comparison model. The superior-subordinate system comparison model is based on the superior-subordinate system comparison model through the processes of system document matching, system document parsing, clause matching, quantitative rule filtering, correlation filtering and quantitative comparison model, so that the big model can judge and answer the audit clues related to the quantitative clauses.

[0078] Specifically, in an embodiment of the present application, the system first receives the pairs of quantitatively related clauses between superiors and subordinates screened by the SentBERT model, and then these pairs of clauses are input into the large model for the comparison of superior and subordinate systems. The large model uses its neural network structure to conduct an in-depth analysis of each pair of clauses to identify the potential relationships and differences between them. This process involves a multi-layer Transformer architecture within the model, in which each layer extracts and transforms different features of the data in order to capture the subtle differences between the clauses.

[0079] For example, in a feasible implementation, the system first receives the superior system documents and subordinate system documents provided by the user through a user interface. The user interface provides a simple and clear interface, where users can easily upload files and start the audit clue generation process. After uploading, the system automatically performs file parsing and clause matching, and then uses the SentBERT model to filter out non-relevant and non-quantitative clause pairs.

[0080] After processing by the SentBERT model, the system will present the screened quantitatively relevant clause pairs to the user. The user can see these clause pairs on the interface and can choose to input them into the institutional comparison model for further analysis. The user only needs to click, and the system will automatically complete the model's reasoning process and display the generated audit clues on the interface.

[0081] Ultimately, the audit clues that users see on the interface are the results of large-scale model analysis. These clues describe in detail the differences between the upper and lower system clauses and potential audit issues. Users can conduct further audit work based on these clues, which greatly improves the efficiency and accuracy of the audit. This process not only reduces the workload of manual review, but also improves the quality of audit results.

[0082] The present embodiment provides a method for auditing system clauses, which obtains pairs of upper and lower clauses by traversing and matching the clauses of the upper and lower system documents and the clauses of the lower and lower system documents obtained based on pre-analysis; performs quantitative correlation filtering on the upper and lower clause pairs through a pre-trained quantitative correlation identification model to obtain upper and lower quantitative correlation clause pairs; inputs the upper and lower quantitative correlation clause pairs into a pre-trained upper and lower system comparison large model to output system audit clues, and after traversing and matching the clauses of the upper and lower system documents through the above method, performs clause quantitative correlation filtering through a pre-trained quantitative correlation model to quickly exclude non-quantitative irrelevant clause pairs, and then inputs the quantitative and relevant clause pairs into the upper and lower system comparison large model, automatically judges all the screened quantitative and relevant clause pairs, and batch outputs audit clues between the upper and lower system clauses, thereby significantly improving the accuracy and efficiency of system clause audits and reducing audit costs.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 The training process of the quantitative correlation identification model further includes steps A1000 to A4000:

[0084] Step A1000: Obtaining upper-lower level training clause pairs and pre-constructed quantization rule expressions;

[0085] It should be noted that in the embodiment of the present application, in order to efficiently screen out all quantitative and relevant terms for automatic judgment, a SentBERT quantitative correlation model is trained to achieve rapid filtering of relevant terms, including the steps of "quantitative data annotation system", "training SentBERT quantitative correlation model", "testing, evaluation and optimization", and "reasoning and optimization".

[0086] The system first collects a large amount of training data from superior and subordinate institutional documents, which contain a series of clauses. At the same time, the system also uses a series of pre-defined quantitative regular expressions, which are designed to help the system better understand and identify key quantitative information in clauses. Quantitative regular expressions usually include a series of patterns and keywords that can guide the system to identify quantitative-related information in clauses.

[0087] In addition, it should be noted that the purpose of this step is to establish a preliminary data set, which will be used in the subsequent training process. Through this data, the system can learn how to identify and process quantitative information, thereby improving the accuracy and efficiency of the audit. The system first extracts the clauses of the superior and subordinate system documents from the database, and then extracts the quantitative regular expressions through a series of text processing and pattern matching techniques. These expressions include the definition of quantitative information such as time, quantity, frequency, etc., which provides a basis for the subsequent quantitative correlation annotation.

[0088] Specifically, after analyzing the system clauses, in this embodiment, a set of quantitative data annotation system is developed according to the characteristics of some quantitative data, such as Figure 4 and Figure 5 As shown, Figure 4 This is a schematic diagram of the automatic annotation process of the quantitative correlation of the upper and lower clauses of the clause audit method of this application system. Figure 5 This is a schematic diagram of the BIO automatic annotation process for quantifiers in the upper and lower level clauses of the application system clause audit method.

[0089] By running the pre-set python program for automatic labeling of the quantitative correlation of superior and subordinate terms, the program will read the superior and subordinate terms in eXcel, one column of which is "superior terms" and one column of "subordinate terms". After the program is finished, a column of "correlation" results will be added to eXcel. Or run the python program for automatic labeling of the quantified words in the superior and subordinate terms in BIO. After the program is finished, two columns of "labeled_superior terms" and "labeled_subordinate terms" results will be added.

[0090] The quantitative data annotation system is used to automatically annotate the hierarchical system, and then manually proofread the annotations and test the data. The accuracy of this set of annotations is 95%, saving a lot of time for manual annotation.

[0091] In this embodiment, the quantization rule expression includes a word-removing quantization rule expression and a related rule expression, wherein the word-removing quantization rule expression is also Rule 1: remove certain keywords containing numbers in the sentence, such as "Article 1, Article 2, Article 3...Article 10, Section 1, Section 2...Section 10, Item 1, Item 2...Item 10, (I), (II), (III)...(Ten), zero group, one type, two copies, three levels, four items, five parties, six categories, seven collections, eight clauses, nine is, ten parts, one aspect, second review number, Annex 0, Annex 1, Annex 2, Annex 3... Annex 0, Annex 1, Annex 2, Annex 3...match the year and month format, such as April 2024, match the month and day format, such as April 15, match the year, month and day format, such as April 15, 2024, in XXXX, from XX day, revised on XX day, further, match numbers plus periods, such as 1., 2., 3., etc., match numbers in brackets plus brackets , such as (1), (2), (3), match the numbers in brackets and add brackets, such as (1), (2), (3), consistent, both, this responsibility, add one, some, further, should be taken, certain, this point, one certificate and five numbers, third party, single source, one-time, unified, general, once confirmed, any, one, reach an agreement, the first time, last quarter, the same, No. 534, avoid one or two levels, one or two categories, one, one size fits all, submit together, three certificates in one, carbon dioxide, three synchronizations, only, one of the circumstances, consistent content, together, a period of time, the two cannot be obtained at the same time, one of the circumstances, MD5, one, only, one, one matter, one discussion, all, one-time, one, single source, only, one by one, 2G, 3G, 4G, 5G, three-party signature, three inspections, one car and one bill, one bill and one settlement, second, fourth, fifth, 10086, 10010, 4001100868, 12315..."

[0092] The relevant rule expressions include Rule 2 and Rule 3. Rule 2 is used to find whether the clause contains zero, one, two, three, four, five, six, seven, eight, and nine. Rule 3 is used to find whether the clause contains keywords such as not less than, not more than, insufficient, at least, at most, should be, less than, more, not allowed, not allowed, prohibited, cannot, must, should not, not less than, not higher than, not exceeding, not less than, must not be higher than, how long, deadline, time limit, first half of the year, second half of the year, every year, every six months, every quarter, every month, every day, etc.

[0093] Based on the above rules, corresponding regular expressions can be generated, such as rule 2 and rule 3, corresponding to rule2 = re.search(r'\d|zero|one|two|three|four|five|six|seven|eight|nine', stripanswer), and rule3 = re.search(r'not less than|not more than|less than|at least|at most|should be|less than|more than|not allowed|not allowed|prohibited|cannot|must not|should not|not less than|not higher than|not more than|not less than|must not be higher than|how long|deadline|time limit|first half of the year|second half of the year|every year|every half year|every quarter|monthly|daily', stripanswer).

[0094] Step A2000: According to the quantization rule expression, the upper and lower training clause pairs are quantized and labeled with quantization words to obtain the upper and lower quantized related training clause pairs;

[0095] It should be noted that the system will use the above quantitative rule expressions to conduct detailed analysis and annotation of the upper and lower training clause pairs. This process involves an in-depth understanding of each clause statement and the identification of the quantitative information therein. The system will mark those keywords and expressions related to quantification, thereby providing rich annotation data for subsequent model training.

[0096] In addition, it should be noted that the purpose of this step is to create a training data set with quantitative relevance annotations, which will be used to train the quantitative relevance identification model. Through these annotations, the model can learn how to identify and process quantitative information, thereby improving the accuracy of the audit. For example, in a specific implementation, all the data of the superior and subordinate clauses are automatically read through a pre-set python program, and they are processed one by one. First, the original superior and subordinate clauses are pieced together into a sentence, and then cleaned to remove spaces, line breaks, etc. Then, according to the above rule one, certain fixed keywords are removed from the sentence. Then, it is determined whether the superior and subordinate clauses without certain fixed keywords satisfy both rules two and three. If so, the pair of superior and subordinate clauses is marked as "related", otherwise it is marked as irrelevant.

[0097] The quantifier tagging adopts similar steps, using a written python program to automatically read all the subordinate clauses and process them one by one. First, the original subordinate clauses are cleaned separately, and spaces and line breaks are removed. Then, according to rule one, some fixed keywords are removed from the sentence. After that, the processed subordinate clauses are segmented separately. Next, the keywords in rule two and rule three are matched, and BIO is used for tagging. The keyword is marked with _B_li at the beginning, _I_li in the middle and at the end, and _O is marked for the remaining words in the original subordinate clauses.

[0098] Step A3000: iteratively training a pre-built quantization correlation identification initial model based on the upper and lower level quantization correlation training clauses, and obtaining clause correlation results;

[0099] It should be noted that the system can use the upper and lower quantitative correlation training clause pairs to train the quantitative correlation identification model. This process involves iterative training of the model, in which the model continuously learns and optimizes to improve its ability to identify quantitative correlations. In this way, the model can gradually improve its accuracy until a satisfactory level is reached. In addition, it should be noted that the purpose of this step is to train a model that can accurately identify quantitative correlations. This model will be used in subsequent audit processes to help the system quickly identify and process quantitative information. In this embodiment, a quantitative correlation identification model can be initialized first, and then the model can be iteratively trained using the upper and lower quantitative correlation training clause pairs. During the training process, the model continuously adjusts its parameters to improve its ability to identify quantitative correlations. Through multiple iterations, the model can eventually accurately identify the quantitative correlations in the upper and lower clauses.

[0100] More specifically, the SentBERT model is an optimized model of the BERT model, which is specifically used to generate sentence-level embedding representations. Through fixed-size sentence embedding, optimized network structure, fine-tuning mechanism, and efficient similarity comparison, the SentBERT model is more suitable for processing text classification tasks. SentBERT performs well in these tasks. The model structure is as follows: Figure 5 shown.

[0101] Step A4000: If the relevance identification accuracy corresponding to the clause relevance result meets the preset accuracy threshold, the quantitative relevance identification model is obtained.

[0102] It should be noted that the system will evaluate the performance of the quantitative relevance identification model, especially its accuracy. This process involves statistics on the data that the model identifies correctly and the data that the model identifies incorrectly, and then calculates the accuracy and recall of the model based on these data. If the accuracy of the model meets the preset threshold, then the model will be considered valid and can be used in subsequent audit processes. In addition, it should be noted that the purpose of this step is to ensure the quality and reliability of the quantitative relevance identification model. By setting a preset accuracy threshold, the system can ensure that only those models whose performance meets the requirements can be used in actual audit work, thereby ensuring the accuracy of the audit results.

[0103] For example, in a specific implementation, the system first tests the quantitative relevance identification model, and then counts the data that the model identifies correctly and incorrectly. Based on these data, the system calculates the accuracy and recall of the model. If the accuracy of the model exceeds the preset threshold, the system will save the model as a quantitative relevance identification model for subsequent audit work. This process ensures the quality and reliability of the model and provides strong support for audit work.

[0104] More specifically, the superior-subordinate clauses labeled in the previous step can be used to train the SentBERT quantitative correlation model, that is, to obtain the "labeled_superior clause", "labeled_subordinate clause" and "correlation" column data. The training data is shown in Table 1. The superior-subordinate system clauses are annotated with BIO, and contain quantitative keywords, such as "at least" and "monthly", which are annotated with B_li, I_li, and other words are annotated with _O. The first piece of data, both the superior and subordinate systems, contain quantitative keywords, and they also describe the same thing, so all the correlations are "related". Although the second piece of data, the superior and subordinate system, contains quantitative keywords, it does not describe things in the same field, so it is marked as irrelevant. If the superior and subordinate systems do not contain quantitative keywords, or do not describe things in the same field, they are also marked as irrelevant.

[0105] Table 1. Examples of training data

[0106]

[0107] In a feasible implementation, the upper and lower training clause pairs include an upper clause sentence and a lower clause sentence, the quantization regular expression includes a word-removing quantization regular expression and a related regular expression, and step A2000 may include steps A2100 to A2500:

[0108] Step A2100: traverse the upper and lower level training clause pairs, and splice the upper level clause sentences and the corresponding lower level clause sentences into upper and lower level clause sentences;

[0109] Step A2200: removing fixed keywords of the upper and lower clauses according to the word removal quantification rule expression to obtain a clean sentence of the upper and lower clauses;

[0110] Step A2300: judging whether the upper and lower clause clean statements are related clause statements according to the related rule expressions;

[0111] Step A2400: In response to the upper-lower-lower clause clean sentence being a related clause sentence, marking the upper-lower-lower clause pair corresponding to the upper-lower-lower clause clean sentence as an upper-lower-lower related training clause pair;

[0112] Step A2500: Match and mark the target quantification words in the upper-lower level related training clause pairs according to the related regular expressions to obtain the upper-lower level quantification related training clause pairs.

[0113] It should be noted that the construction of the upper and lower training clause pairs is the basis of the quantitative relevance identification model training process. In this process, the system first traverses all the upper clause sentences and the corresponding lower clause sentences, and splices them into upper and lower clause sentences. The purpose of this step is to create a complete sentence pair so that the quantitative information can be more accurately identified and labeled later.

[0114] In addition, it should be noted that the word removal quantification regular expressions play a key role in this process. The system uses these regular expressions to remove fixed keywords in the upper and lower clauses, thereby obtaining clean sentences of the upper and lower clauses. The purpose of this step is to eliminate interference factors that do not affect the quantitative relevance judgment, so that the model can focus more on identifying and quantifying relevant information.

[0115] It should be noted that the use of relevant regular expressions is the key to determining whether the upper and lower clauses are related clauses. The system uses these regular expressions to determine whether the upper and lower clauses are related. The purpose of this step is to identify those lower-level clauses that are related to the upper-level system in terms of content and quantitative information.

[0116] If the clean sentence of the upper and lower clauses is judged as a related clause sentence, the system will mark the corresponding upper and lower training clause pair as the upper and lower related training clause pair. The purpose of this step is to prepare high-quality related annotation data for model training to ensure that the model can accurately learn the features of quantitative correlation.

[0117] Finally, the system matches the target quantifiers in the upper-lower level related training clause pairs according to the relevant regular expressions and annotates them to obtain the upper-lower level related quantification training clause pairs. The purpose of this step is to further enrich the annotation information of the training data so that the model can not only identify the quantification correlation, but also accurately locate and understand the quantification information in the clauses.

[0118] For example, in a feasible implementation, the system first extracts the clauses of the superior and subordinate system documents from the database, and then splices them into superior and subordinate clauses through text processing technology. Next, the system applies word removal quantification rule expressions to remove non-critical words in the clauses, such as clause numbers, common phrases, etc., to obtain clean clauses of superior and subordinate clauses. Subsequently, the system uses relevant regular expressions to analyze these clean sentences to determine whether they match the relevant quantitative information of the superior system. For matching clause pairs, the system marks them as superior and subordinate related training clause pairs, and further identifies and marks the quantitative keywords therein, such as numbers, time frequencies, etc. In this way, the system can generate a high-quality superior and subordinate quantitative related training clause pair data set, which provides rich annotation information for the training of the quantitative correlation identification model. In this way, the system can train a model that can accurately identify and process quantitative information, thereby improving the efficiency and accuracy of audit work.

[0119] In a feasible implementation manner, step A3000 may include steps A3100 to A3300:

[0120] Step A3100: Perform sentence embedding and pooling processing on the upper and lower level quantization related training clause pairs to obtain the upper level quantization clause sentence embedding vector and the lower level quantization clause sentence embedding vector;

[0121] Step A3200: Calculate the difference vector between the upper and lower quantified clause sentences based on the upper and lower quantified clause sentence embedding vectors;

[0122] Step A3300: Perform probability distribution conversion based on the difference vector of the upper and lower level quantitative clause statements to obtain the clause relevance result.

[0123] It should be noted that in this embodiment, the training terms related to the upper and lower level quantization obtained in the above steps can be divided into a training set, a validation set, and a test set in a ratio of 7:1:2, that is, 7 sets of data ensure the effect of model learning, 1 set of data is used to verify the effect of the model during the training process, and the model is continuously optimized and adjusted, and 2 sets of data are used to verify the performance of the model on unseen data sets. The above training set data is input into the SentBERT model for training.

[0124] During the training process of the SentBERT model, the first step is to encode the sentence. This step takes advantage of the BERT model's ability to convert the input text into vector representations in a high-dimensional space that can capture the semantic information of the sentence. Through its deep Transformer architecture, the BERT model is able to understand the contextual relationships in the text, resulting in a richer and more accurate embedding representation.

[0125] After encoding, the model processes the sentence's embedding representation through a pooling layer. Pooling operations, such as mean pooling, can combine the embedding representations of all words in a sentence into a fixed-length vector that represents the semantic information of the entire sentence. In this process, the embedding vectors of each word in the sentence are aggregated to obtain a comprehensive vector that can represent the entire sentence.

[0126] The model then calculates the similarity or difference between the two sentence embedding vectors. By calculating the modulus of the difference between the two vectors, i.e. |uv|, the model is able to obtain a vector representing the difference between the two sentences. This vector reflects the distance between the two sentences in the semantic space. The smaller the distance, the more similar the two sentences are; the larger the distance, the greater the difference between the two sentences.

[0127] Finally, the model inputs these difference vectors into a linear layer and undergoes a softmax operation to convert the numerical values ​​into a probability distribution. This probability distribution can represent the probability of two sentences being related or unrelated. In this way, the model can classify the two input sentences and determine whether they are semantically related.

[0128] By training on large amounts of data and continuously optimizing model parameters, the SentBERT model is able to judge quantitatively relevant pairs of terms more and more accurately. During training, the model learns which semantic features and patterns are related to quantitative relevance, thereby improving its accuracy in practical applications. Ultimately, choosing the model with the best preservation effect means choosing the model that performs best on the validation set, which can provide the most reliable relevance judgment in practical applications.

[0129] After this step of model training, a SentBERT quantitative correlation model is obtained. When two upper and lower terms are input, the model can determine whether they are related and output "related" or "not relevant".

[0130] In a feasible implementation manner, before step A4000, steps A3400 to A3800 are also included:

[0131] Step A3400: Obtain the clause relevance result;

[0132] Step A3500: Counting the model recognition correct data and model recognition incorrect data corresponding to the clause relevance result, wherein the model recognition correct data includes relevant recognition correct data and irrelevant recognition correct data, and the model recognition incorrect data includes relevant recognition incorrect data and irrelevant recognition incorrect data;

[0133] Step A3600: Calculating the model accuracy based on the relevant identified correct data and the irrelevant identified incorrect data;

[0134] Step A3700: Calculating a model recall rate based on the relevant correctly identified data and the relevant incorrectly identified data;

[0135] Step A3800: Calculate the accuracy of the correlation identification corresponding to the clause correlation result based on the model accuracy and model recall.

[0136] It should be noted that in the training process of the quantitative relevance identification model, obtaining the results of the relevance of terms is a key step in evaluating the performance of the model. This step involves obtaining the model's prediction results from the validation set and comparing them with the actual relevance labels. By comparing, the number of relevant and irrelevant terms pairs correctly identified by the model, as well as the number of relevant and irrelevant terms pairs incorrectly identified, can be determined. These data provide the basis for calculating the model's precision and recall.

[0137] In addition, it should be noted that the accuracy and recall of the model are two important indicators for measuring model performance. Accuracy reflects the proportion of correct predictions made by the model, while recall reflects the proportion of relevant terms pairs identified by the model to all actual relevant terms pairs. The calculation of these two indicators is based on the data correctly identified by the model and the data incorrectly identified by the model. Through these two indicators, the performance of the model in the task of quantitative relevance judgment can be comprehensively evaluated.

[0138] In this embodiment, the system first uses a pre-built quantitative rule expression to annotate the upper and lower level training clause pairs to generate quantitative related training clause pairs. Then, by iteratively training the SentBERT quantitative correlation identification initial model, the clause correlation result is obtained. After obtaining the clause correlation result, the system statistical model identifies correct and incorrect data, including relevant identification correct data, irrelevant identification correct data, relevant identification incorrect data, and irrelevant identification incorrect data. Based on these data, the system calculates the accuracy and recall of the model, and further calculates the correlation identification accuracy. If the correlation identification accuracy meets the preset accuracy threshold, the model is saved as a quantitative correlation identification model for subsequent quantitative correlation judgment tasks.

[0139] For example, in a feasible implementation, the system first obtains a large number of superior-subordinate clause pairs from the training set, and uses quantitative regular expressions to annotate these clause pairs to generate quantitatively relevant training clause pairs. Then, the system inputs these annotated data into the SentBERT model for iterative training. During the training process, the system continuously adjusts the model parameters to improve the model's ability to identify quantitative correlations. After the training is completed, the system evaluates the performance of the model on the validation set, and statistically identifies correct and incorrect data by comparing the model's prediction results and actual labels. Based on these data, the system calculates the accuracy and recall of the model, and combines these two indicators to calculate the accuracy of correlation identification. If the accuracy of correlation identification meets the preset threshold, the system saves the model as a quantitative correlation identification model for the actual system document audit clue automatic generation task. This process ensures that the model has high accuracy and recall in the quantitative correlation judgment task, thereby improving the quality and efficiency of audit clue generation. After the model training is continuously tuned, the accuracy of the SentBERT quantitative correlation model is evaluated on the test data set, and the evaluation adopts the method of P value, R value, and F1 value.

[0140]

[0141]

[0142] Among them, TP represents the number of positive examples (relevant) correctly identified as positive examples by the system, FP represents the number of negative examples (irrelevant) incorrectly identified as positive examples by the system, FN represents the number of positive examples (relevant) that the system failed to correctly identify as positive examples, and TN represents the number of negative examples (irrelevant) correctly identified as negative examples by the system.

[0143] In a feasible implementation manner, before step S2000, steps S1100 to S1300 may also be included:

[0144] Step S1000: generating a text array of upper-lower-level quantification clause pairs based on the upper-lower-level quantification clause pairs;

[0145] Step S1000: performing encoding conversion on the text array based on the upper and lower quantization clauses to obtain normalized upper and lower quantization clause pair sentence embeddings, wherein the normalized upper and lower quantization clause pair sentence embeddings include normalized upper quantization clause sentence embeddings and normalized lower quantization clause sentence embeddings;

[0146] Step S1000: Calculate the sentence embedding dot product between the normalized upper-level quantization clause sentence embedding and the normalized lower-level quantization clause sentence embedding to obtain the sentence embedding dot product result of the upper- and lower-level quantization clause pair;

[0147] Step S2000 may include step S2100:

[0148] Step S2100: Performing quantitative correlation filtering on the sentence embedding dot product results of the superior-subordinate clause pair through a pre-trained quantitative correlation identification model to obtain superior-subordinate quantitative correlation clause pairs.

[0149] It should be noted that in the embodiments of the present application, in order to speed up the running time of model reasoning, data parallel processing is used to reduce I / O operations, achieve memory optimization, and improve reasoning speed.

[0150] First, the system generates a text array of superior-subordinate quantitative clause pairs and converts these clause pairs into a format that can be processed by the model. This step ensures that the data for subsequent processing is structured and can be understood by the model.

[0151] Next, the system encodes and converts these text arrays to obtain normalized sentence embeddings of the upper and lower quantitative clause pairs, converting the text data into numerical representations. The normalization process is to eliminate the impact of texts of different lengths and sizes on the calculation process, ensuring that the model can fairly compare and analyze each pair of clauses. Then, the system calculates the sentence embedding dot product between the normalized upper quantitative clause sentence embedding and the normalized lower quantitative clause sentence embedding. The dot product calculation is a method to measure the similarity of two vectors, which can help the system understand the semantic similarity or difference between two clauses.

[0152] Finally, the system can perform quantitative correlation filtering on the dot product results of the embedding of the superior-subordinate clause pairs through the pre-trained quantitative correlation identification model. In the above method, by converting the batch of superior-subordinate quantitative clause pairs into text pair arrays in advance, and then calculating the dot product of the embedding vectors between each text pair, the quantitative correlation model, that is, the superior-subordinate system comparison model, can save the computing power consumption of the above steps and directly output the model based on the vector dot product, thereby improving the computing efficiency.

[0153] For example, by defining a SentenceSimily class function, initialize the fine-tuned SentBERT model, such as through the func:__init__ function, batch load the text array of the upper and lower clauses to be predicted, such as an Excel table, which can be implemented through the func:load_data function, and convert the input upper and lower clause text numbers to be predicted into sentence embeddings through encoding, such as through func:encode. This step is mainly used to convert text into a numerical representation that can be used to calculate the similarity between texts. Finally, return the dictionary res, which contains each unique non-empty text in the input array and its corresponding normalized sentence embedding, and then batch dot product calculates the value of the two encoded sentences, such as through the func:call function. Finally, load the data, encode the data for sentences, calculate the results, and finally save the results, such as through the func:__call__ function. You can also implement the SentenceSimily class to achieve reasoning optimization.

[0154] In the embodiments of this application, please refer to Figure 6 The training process of the upper-lower level system comparison large model includes steps B1000 to B5000:

[0155] Obtaining a superior-subordinate quantitative clause audit clue data set and a pre-set business scenario prompt, wherein the superior-subordinate quantitative clause audit clue data set includes superior-subordinate quantitative clauses and corresponding superior-subordinate quantitative clause audit clues;

[0156] Step B1000: Generate a large model training input sentence based on the upper and lower level quantitative terms and business scenario prompts;

[0157] Step B2000: training a pre-built initial large model based on system comparison based on the large model training input sentence to obtain a large model based on system comparison training;

[0158] Step B3000: Repeat data training and parameter optimization according to the large model based on system comparison training, and obtain the training model output result;

[0159] Step B4000: If the degree of fit between the training output result and the audit clues of the superior-subordinate quantitative clauses is greater than the preset fit threshold, the superior-subordinate system comparison model is obtained.

[0160] It should be noted that the training process of the big model for comparing the upper and lower systems is a systematic method that aims to improve the efficiency and accuracy of system document audits through deep learning technology. This process involves extracting quantitative clauses from a large number of system documents and using this data to train a big model that can understand and compare upper and lower system documents.

[0161] First, we need to obtain the audit clues dataset of quantitative clauses of superiors and subordinates. These datasets contain the specific clauses and corresponding audit clues in the superior and subordinate system documents. These datasets are obtained by parsing the system documents and filtering the quantitative rules, ensuring the quality and relevance of the data. At the same time, pre-set business scenario prompts, such as audit instructions or problem descriptions, will be used as part of the model input to help the model better understand the specific needs of the audit.

[0162] Next, we generate input sentences for large model training based on the upper and lower level quantitative clauses and business scenario prompts. The purpose of this step is to build an input data set that can reflect actual audit scenarios, so that the model can learn how to generate accurate audit clues based on specific business scenarios and institutional clauses during the training process.

[0163] Subsequently, these training input sentences are used to train the pre-built initial large model based on system comparison. In this process, the model will learn how to understand and compare the clauses in the upper and lower system documents and generate audit trails based on these clauses. Through repeated data training and parameter optimization, the performance of the model will be continuously improved, and finally a well-trained large model will be obtained.

[0164] Finally, evaluate the fit between the output of the training model and the audit clues of the upper and lower level quantitative clauses. This step is completed by comparing the audit clues generated by the model with the actual audit clues. If the fit is greater than the preset threshold, it means that the model has a high degree of accuracy and practicality and can be used in actual system document audit work.

[0165] For example, in a feasible implementation, the system first extracts quantitative clauses of superiors and subordinates from a massive institutional document library, and generates a series of training input sentences in combination with pre-set business scenario prompts. Then, the system uses these input sentences to train a large model based on the Transformer architecture. Through multi-layer QwenBlock and attention mechanisms, the model can deeply understand the content of institutional clauses and the relationship between superiors and subordinates. During the training process, the system continuously adjusts the model parameters and optimizes the model structure until the audit clues generated by the model are more consistent with the actual audit clues than the preset threshold. In this way, the system finally obtains a large model that can automatically generate audit clues for institutional documents, significantly improving the efficiency and quality of audit work.

[0166] In addition, it should be noted that in this embodiment, the training system document comparison clue output model can be Qwen14b, or it can be Qwen72b, Llama2-70B and other large language models with hundreds of billions of parameters. The upper-lower system comparison large model understands the two upper-lower clauses and determines whether there is an audit clue problem. It mainly relies on the reasoning ability of the large model. In theory, the larger the model parameters, the more accurate the audit clue output. The training steps of the upper-lower system comparison large model include "data annotation", "prompt design", "training system comparison clue output model", "testing and manual evaluation" and other processes.

[0167] The quantitative comparison model receives audit clue datasets corresponding to the hierarchical system that have been annotated by professional auditors. These datasets have been strictly screened and quality controlled to ensure that the audit clue datasets input into the model meet the requirements of the intelligent audit system SFT. The training data are shown in Table 2.

[0168] Table 2 Examples of clues to superior and subordinate quantitative clauses

[0169]

[0170] In the field of artificial intelligence, prompt is a technique for guiding large language models to generate specific responses. Prompt is usually a text input that describes the beginning of the conversation for the expected model to perform a task. Through carefully designed prompts, the model can better understand the user's intentions and generate outputs that are both useful and expected.

[0171] In order to obtain clues for comparing the superior-subordinate systems that are more in line with the business scenarios, in this embodiment, the data format of the input big model is as shown in Table 3. The business scenario prompt consists of role + question + requirement + superior system + subordinate system. The prompt will be continuously updated according to the model performance effect. The program will splice the business scenario prompt and the superior-subordinate clauses and send them into the big model together.

[0172] Table 3 Large model input example (prompt)

[0173]

[0174]

[0175] Specifically, the large model in this embodiment selects the Qwen14B large model, which is based on the Transformer architecture. The input sentence first enters the embedding layer for encoding. The maximum sequence length supported by the model is 2048 tokens, and the model uses a vocabulary size of more than 150,000. Then there is a multi-layer QWenBlock. The model contains 40 layers, each layer uses 40 attention heads, and the size of each attention head is 5120, which provides sufficient feature extraction capabilities, uses RoPE for relative position encoding, and uses SwiGLU as the activation function. RMSNorm is used for normalization, and finally the final result is output through Linear.

[0176] The LORA method is used to fine-tune the large model. Through model training with large amounts of data and continuous optimization of parameters, the answers of the large model are made more in line with the input audit clue types, ultimately achieving the effect of inputting quantitative data clues of the hierarchical system and outputting them.

[0177] The training data is also divided into training set and test set in the ratio of 7:1:2. The above training set data is input into the large model for training, and the parameters are adjusted using the LORA method.

[0178] After this step of model training, we obtained a large model for comparing superior and subordinate systems. By inputting two superior and subordinate clauses, the model can output quantitative clues to the problems between the two clauses.

[0179] After continuous optimization of model training, the accuracy of the large model of system comparison clues was evaluated on the test data set. Because the output of the upper and lower system clues is a piece of semantics, and the BLEU and ROUGU values ​​evaluated by the machine focus on the accuracy of translation and the completeness of information, manual evaluation is adopted.

[0180] Auditors manually score the test data set to determine whether the model output is correct, scoring from 0-100. For example, 100 points for consistency with the facts and 80 points for 80% correct. After evaluation, the average score of the current 150 data manually evaluated is 85 points, which meets the requirements of system design audit scenarios.

[0181] In this embodiment, the generative big model technology is introduced. Through the processes of system document matching, system document parsing, clause matching, quantitative rule filtering, correlation filtering and quantitative comparison big model, the big model can judge and answer the audit clues related to the quantitative clauses. This model is used to audit whether the system design of some business and management areas of the subordinate company complies with the superior system, reflecting the risks of the system in terms of comprehensiveness, effectiveness and unity. It aims to improve the company's system management level and lay a solid foundation for the compliance development of business.

[0182] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the audit method of the system clauses of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0183] Please refer to Figure 7 The present application also provides a system clause audit device, the system clause audit device comprising:

[0184] A pairing module 71 is used to pair the clauses of the superior system document and the clauses of the subordinate system document obtained by pre-analysis to obtain a pair of superior and subordinate clauses;

[0185] A filtering module 72, configured to perform quantitative correlation filtering on the upper and lower level clause pairs through a pre-trained quantitative correlation identification model to obtain upper and lower level quantitative correlation clause pairs;

[0186] The output module 73 is used to input the superior-subordinate quantitative related clause pairs into the pre-trained superior-subordinate system comparison model to obtain system audit clues.

[0187] The system clause audit device provided by the present application adopts the system clause audit method in the above embodiment, which can solve the technical problems of system clause audit. Compared with the prior art, the beneficial effects of the system clause audit device provided by the present application are the same as the beneficial effects of the system clause audit method provided by the above embodiment, and the other technical features in the system clause audit device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0188] The present application provides a system clause audit device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the system clause audit method in the above-mentioned embodiment one.

[0189] Reference below Figure 5 , which shows a schematic diagram of the structure of a system clause audit device suitable for implementing the embodiment of the present application. The system clause audit device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5The system clause audit device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0190] like Figure 5 As shown, the system clause audit device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the system clause audit device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the system clause audit device to communicate with other devices wirelessly or wired to exchange data. Although the system clause audit device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0191] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0192] The system clause auditing device provided in the present application adopts the system clause auditing method in the above-mentioned embodiment. Compared with the prior art, the beneficial effects of the system clause auditing device provided by the present application are the same as the beneficial effects of the system clause auditing method provided in the above-mentioned embodiment, and other technical features in the system clause auditing device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0193] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0194] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0195] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the system clause audit method in the above-mentioned embodiment.

[0196] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0197] The above-mentioned computer-readable storage medium may be included in the system clause audit device; or it may exist independently without being assembled into the system clause audit device.

[0198] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0199] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0200] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0201] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned system clause audit method, and can solve the technical problems of system clause audit. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the system clause audit method provided in the above-mentioned embodiment, and will not be repeated here.

[0202] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned system clause audit method when executed by a processor.

[0203] The computer program product provided by this application can solve the technical problem of system clause auditing. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the system clause auditing method provided by the above embodiment, which will not be repeated here.

[0204] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for auditing system clauses, characterized in that: The method comprises: Pair the clauses of the superior system document and the clauses of the subordinate system document obtained in advance, and obtain the superior-subordinate clause pair; Performing quantitative correlation filtering on the upper-lower clause pairs through a pre-trained quantitative correlation identification model to obtain upper-lower clause pairs; The pairs of quantitatively related clauses between superiors and subordinates are input into a pre-trained large model for comparing superior-subordinate systems to obtain system audit clues.

2. The method according to claim 1, characterized in that The training process of the quantitative correlation identification model includes: Obtain superior-subordinate training clause pairs and pre-built quantized rule expressions; According to the quantization rule expression, the upper and lower training clause pairs are quantized and labeled with quantization terms to obtain upper and lower quantized related training clause pairs; Iteratively training the pre-built quantization correlation identification initial model based on the upper and lower level quantization correlation training clauses, and obtaining clause correlation results; If the relevance identification accuracy corresponding to the clause relevance result meets the preset accuracy threshold, the quantitative relevance identification model is obtained.

3. The method according to claim 2, characterized in that The upper-lower training clause pair includes an upper-level clause sentence and a lower-level clause sentence, the quantization rule expression includes a word-removed quantization rule expression and a related rule expression, and the step of performing quantization correlation labeling and quantization word labeling on the upper-lower training clause pair according to the quantization rule expression to obtain the upper-lower quantization related training clause pair includes: Traversing the upper and lower training clause pairs, and concatenating the upper clause sentences and the corresponding lower clause sentences into upper and lower clause sentences; According to the word removal quantification rule expression, fixed keywords of the upper and lower clause sentences are removed to obtain the upper and lower clause clean sentences; Determine whether the upper and lower clause clean statements are related clause statements according to the related rule expressions; In response to the upper and lower clause clean sentences being related clause sentences, marking the upper and lower training clause pairs corresponding to the upper and lower clause clean sentences as upper and lower related training clause pairs; According to the relevant regular expressions, the target quantification words in the upper-lower level related training clause pairs are matched and marked to obtain the upper-lower level quantification related training clause pairs.

4. The method according to claim 2, characterized in that The step of iteratively training the pre-built quantization correlation identification initial model based on the upper and lower level quantization correlation training clauses and obtaining the clause correlation results includes: Perform sentence embedding and pooling processing on the upper and lower level quantization related training clause pairs to obtain the upper level quantization clause sentence embedding vector and the lower level quantization clause sentence embedding vector; Calculate the difference vector of the upper and lower quantified clause sentences based on the upper and lower quantified clause sentence embedding vectors; Based on the difference vector of the upper and lower level quantitative clause statements, probability distribution conversion is performed to obtain the clause relevance result.

5. The method according to claim 2, characterized in that If the accuracy of the relevance identification corresponding to the clause relevance result meets the preset accuracy threshold, the step of obtaining the quantitative relevance identification model also includes: Obtain the relevance results of the terms; Counting the model recognition correct data and model recognition incorrect data corresponding to the clause relevance results, wherein the model recognition correct data includes relevant recognition correct data and irrelevant recognition correct data, and the model recognition incorrect data includes relevant recognition incorrect data and irrelevant recognition incorrect data; Calculate the model accuracy based on the relevant identified correct data and the irrelevant identified incorrect data; Calculating a model recall rate based on the relevant correctly identified data and the relevant incorrectly identified data; The relevance identification accuracy corresponding to the clause relevance result is calculated based on the model accuracy and model recall.

6. The method according to any one of claims 1 to 5, characterized in that: The training process of the superior-subordinate system comparison model includes: Obtaining a superior-subordinate quantitative clause audit clue data set and a pre-set business scenario prompt, wherein the superior-subordinate quantitative clause audit clue data set includes superior-subordinate quantitative clauses and corresponding superior-subordinate quantitative clause audit clues; Generate a large model training input sentence based on the superior-subordinate quantitative terms and business scenario prompts; Based on the large model training input sentence, a pre-built initial large model based on system comparison is trained to obtain a large model based on system comparison training; Repeat data training and parameter optimization according to the large model based on system comparison training, and obtain the training model output result; If the degree of fit between the training output result and the audit clues of the superior-subordinate quantitative clauses is greater than a preset fit threshold, the superior-subordinate system comparison model is obtained.

7. The method according to claim 1, characterized in that Before the step of performing quantitative correlation filtering on the upper and lower level clause pairs through the pre-trained quantitative correlation identification model to obtain the upper and lower level quantitative correlation clause pairs, the step further includes: Generate a text array of upper-lower-level quantified clause pairs based on the upper-lower-level quantified clause pairs; Based on the upper and lower quantization clauses, the text array is encoded and converted to obtain normalized upper and lower quantization clause pair sentence embeddings, wherein the normalized upper and lower quantization clause pair sentence embeddings include normalized upper quantization clause sentence embeddings and normalized lower quantization clause sentence embeddings; Calculate the sentence embedding dot product between the normalized upper-level quantization clause sentence embedding and the normalized lower-level quantization clause sentence embedding to obtain the sentence embedding dot product result of the upper-level quantization clause pair; The step of performing quantitative correlation filtering on the upper and lower level clause pairs through the pre-trained quantitative correlation identification model to obtain the upper and lower level quantitative correlation clause pairs comprises: The sentence embedding dot product results of the superior-subordinate clause pairs are subjected to quantitative correlation filtering through a pre-trained quantitative correlation identification model to obtain superior-subordinate quantitative correlation clause pairs.

8. A system clause audit device, characterized in that: The device comprises: A matching module is used to match the clauses of the superior system document and the clauses of the subordinate system document obtained in advance, so as to obtain a pair of superior and subordinate clauses; A filtering module, used for performing quantitative correlation filtering on the upper and lower clause pairs through a pre-trained quantitative correlation identification model to obtain upper and lower quantitative correlation clause pairs; The output module is used to input the quantitative related clause pairs between superiors and subordinates into a pre-trained superior-subordinate system comparison model to obtain system audit clues.

9. A system clause audit device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the system clause audit method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the system clause audit method as described in any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the system clause audit method according to any one of claims 1 to 7 are implemented.

Images