Project material consistency verification method and system based on field-level information extraction

Abstract

The application provides a project material consistency verification method and system based on field-level information extraction, relates to the field of clinical research informatization management, and comprises the following steps: receiving multiple project materials to perform format normalization processing; establishing a statistical design parameter registration library; performing field-level information extraction according to key fields and field roles; matching identity information in the value of a dependent document field with the identity of a presence document; comparing a document set with a material list; disambiguating the field value; comparing the representative values of the same field roles in a document pair to obtain a cross-file consistency comparison result; calling a backstepping method from the statistical design parameter registration library to obtain an allowed value range of the total number of enrolled cases, comparing the total number of enrolled cases in the field value with the allowed value range to obtain a sample size closed verification result, and generating a consistency verification result. The application improves the verification efficiency and the traceability of the result in the multi-document association review scene.

Description

Technical Field

[0001] This invention relates to the field of information management technology for clinical research, and in particular to a method and system for verifying the consistency of project application materials based on field-level information extraction. Background Technology

[0002] Clinical research project approval involves multiple interconnected and mutually referencing electronic documents, including the project application, clinical research protocol, informed consent form, statistical analysis plan, and ethics approval document. Because these documents are not created simultaneously or written by the same person, common problems in practice include asynchronous version iterations, inconsistent field descriptions, and scattered recording of statistical design parameters. For example, the protocol version declared in a subordinate document may not exist in the submitted materials; the same number of enrolled cases may be misjudged as contradictory in different documents due to their different roles; and the number of enrolled cases declared in the protocol may appear consistent with the statistical design parameters, but the test methods and endpoint types may not match. Currently, these issues are mainly addressed through manual page-by-page verification, which is inefficient, prone to omissions, and difficult to pinpoint the specific source documents and processes of inconsistencies.

[0003] Existing general contract review systems or project document review platforms mostly use a combination of rule engines and key field extraction to achieve element comparison. Their review logic usually manifests as direct equivalence judgment or threshold judgment after field extraction, without distinguishing between document reference dependencies and version evolution relationships, or considering the need to handle the same terminology differently in different application scenarios. Directly applying such a general framework to multi-document association review scenarios can easily lead to problems such as version iterations being misjudged as errors and incomplete submission materials being silently missed, making it difficult to meet the actual needs of review scenarios with high requirements for document reference relationships and numerical logic.

[0004] Chinese invention patent application number 202411198857.9 discloses an artificial intelligence-based document verification method and system. It uses a document format parsing library to convert documents of different formats into a unified TXT text format and performs structured processing. It supports conversion between multiple document formats and can calculate a document's sensitivity risk score by comprehensively analyzing document text feature vectors, source data feature vectors, and anomaly scores. Based on historical data and a dynamic adjustment mechanism, it calculates a highly adaptable risk threshold. However, it does not perform correlation verification on multiple documents with citation and reference relationships, making it prone to omissions or misjudgments in verification of the same numerical field in different documents and contexts. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for verifying the consistency of project application materials based on field-level information extraction, in order to solve the problems in the prior art that it is impossible to perform correlation verification on multiple documents with reference and basis relationships, and that verification omissions and misjudgments are easily caused by role differences or inherent numerical logic relationships on the same numerical field, thereby improving the verification efficiency and traceability of results in multi-document association review scenarios.

[0006] The technical solution of this invention is implemented as follows: On the one hand, this invention provides a method for verifying the consistency of project initiation materials based on field-level information extraction, including: S1. Receive multiple electronic documents of project approval materials to be verified, normalize the format of the electronic documents, and identify the type of each document based on the document content characteristics to obtain a set of documents labeled with document types; document types include proposals and statistical analysis plans. S2. Set up material lists, identification rules, and key fields to be extracted according to document type, and establish a statistical design parameter registration database; The material list includes missing criteria and completeness criteria. The missing criteria are used to determine whether a document type is not submitted and whether it is recorded as missing from the list. The completeness criteria are used to determine whether the missing document type will cause the comparison between other documents to be blocked. The identity recognition rules stipulate that: documents in the document collection are divided into present documents and subordinate documents according to whether their version identity can be directly determined based on their own field values. The rules also specify the identification method for the present documents and the method for extracting the identity information of the declared document from the field values ​​of subordinate documents. The key fields are set in a three-level hierarchy: document type, field name, and field role. Field roles are used to identify semantically corresponding and comparable fields in different documents. S3. For each document in the document collection, perform field-level information extraction according to key fields and field roles to obtain field values; based on identity recognition rules, match the identity information in the field values ​​of the subordinate documents with the identities of the documents present to obtain matching results; S4. Compare the document set with the material list to obtain the list verification results; S5. Disambiguate the field values ​​under the same field role in the same document to obtain representative values; generate document pairs for cross-file comparison based on the matching results, compare the representative values ​​of the same field role in the document pairs to obtain cross-file consistency comparison results; S6. Based on the statistical design parameters in the field values, retrieve the reverse calculation method from the statistical design parameter registry to obtain the allowable range of the total number of cases enrolled. Compare the total number of cases enrolled in the field values ​​with the allowable range to obtain the sample size closure verification result. S7. Summarize the list verification results, cross-document consistency comparison results, and sample size closure verification results to generate consistency verification results.

[0007] Based on the above technical solutions, preferably, step S4 specifically includes: Document types marked as mandatory in the materials list that do not exist in the document collection will be recorded as missing documents in the list verification results. Document types marked as optional submission in the material list and not present in the document set are not considered missing from the list, and the verification continues. The verification results will indicate that the type was not submitted and the default basis used. The default basis refers to other submitted document types or their field values ​​used in the verification when the document type was not submitted. Document types that are marked as missing in the material list and that prevent the comparison of other documents, and which do not exist in the document set, will be marked as preventing cross-document consistency comparison in the list verification results.

[0008] Based on the above technical solutions, preferably, the matching of the identity information obtained from the subordinate document field with the identity of the present document specifically includes: If no identity information is found in the field values, the matching result will be recorded as undeclared. When a field value contains identity information, and the document collection contains a document whose identity matches the identity information, the matching result is recorded as "declared and present". If a field value contains identity information, and there is no document in the document collection that matches the identity information, the matching result is recorded as "declared but not present".

[0009] Based on the above technical solutions, preferably, the step of generating document pairs for cross-file comparison based on the matching results specifically includes: When the matching result is "declared and present", the document declaring the identity information is paired with the present document that matches the identity; if at least two documents are matched with the same present document as "declared and present", then these documents are further paired with each other to form document pairs. When the matching result is undeclared, and there is only one document present in the document collection identified according to the identity recognition rules, the document with no identity information in the field value is paired with the present document to form a document pair; If the matching result is a declaration but not present, no document pair is generated, and the matching result is recorded in the cross-file consistency comparison result.

[0010] Based on the above technical solutions, preferably, the field roles include the total number of cases included in the group, the number of evaluable cases, the number of completed cases, and the number of cases in each group. The disambiguation of field values ​​under the same document and the same field role specifically includes: Group the field values ​​under the same document and the same field role into the value set of the field role; When all values ​​in the set are the same, the value is used as the representative value. When there are different values ​​in the value set, the field role does not output the representative value; Field roles whose values ​​have been output will be used to compare the same field roles in document pairs; field roles whose values ​​have not been output will not be compared in document pairs.

[0011] Based on the above technical solutions, preferably, the comparison of representative values ​​of the same field roles in the document pair specifically includes: After obtaining the representative values, the representative values ​​of the total number of cases in the group are summed and compared with the representative values ​​of the number of cases in each group; If the sum of the number of cases in each group is not equal to the total number of cases in the group, it is determined that the relationship between roles is inconsistent, and the total number of cases in the document and the number of cases in each group are not included in the comparison of document pairs. The representative values ​​of the same field under the same role in the document pair that were not judged as having inconsistent relationships between roles are compared one by one. Inconsistent values ​​are recorded in the cross-file consistency comparison results.

[0012] Based on the above technical solutions, preferably, the statistical design parameters include endpoint type, hypothesis type, allocation ratio, test direction, test method, and dropout rate; the document types in the document set include protocol documents and statistical analysis plans; the method for retrieving the back-calculation method from the statistical design parameter registry specifically includes: Based on the endpoint type, hypothesis type, allocation ratio, test direction and test method in the field values, determine and retrieve the corresponding reverse calculation method from the statistical design parameter registry; When the statistical design parameters of the statistical analysis plan are complete, reverse calculation is performed using the statistical design parameters of the statistical analysis plan and the total number of cases to be included in the field values ​​of the protocol document; If the proposed design and the statistical analysis plan give different values ​​for the same statistical design parameter, the difference will be recorded in the cross-document consistency comparison results; otherwise, the parameter in the statistical analysis plan will still be used. If there is no statistical analysis plan in the document collection and the statistical design parameters in the scheme document are complete, the reverse calculation is performed using the statistical design parameters in the scheme document; if the statistical design parameters are incomplete, or if there is no corresponding reverse calculation method in the statistical design parameter registration database, the reverse calculation method is not invoked.

[0013] Based on the above technical solutions, preferably, the allowable range of values ​​for obtaining the total number of enrolled cases includes: The statistical design parameters were processed using a reverse calculation method to obtain a continuous solution that was not rounded down or amplified by the dropout rate. For continuous solutions, perform two separate operations: first round up and then amplify by the dropout rate, and second amplify by the dropout rate and then round up, to obtain two sets of integer results. The smaller of the two integer results is taken as the lower limit of the allowable range. The interval width is determined by the dropout rate, which is the difference between two divided by one and the dropout rate. The upper limit of the allowable range is the sum of the lower limit and the interval width minus one. The total number of enrolled cases is compared with the allowable range. If the total number of enrolled cases falls within the allowable range, it is recorded as a sample size closure check result and marked as closed. If it does not fall within the allowable range, it is recorded as a sample size closure check result and marked as a sample size design not closed.

[0014] Furthermore, step S7 specifically includes: The results of the checklist verification, cross-document consistency comparison, and sample size closure verification are categorized by type. The classification results are sorted according to a preset priority, and a verification text is generated to obtain the consistency verification result. The manual annotations are used to confirm, identify false alarms or correct errors in the consistency verification results. Based on these manual annotations, the database of identity recognition rules, key fields or statistical design parameters is supplemented. Re-execute field-level information extraction and comparison on the document collection, compare the verification results before and after the supplementation, and use manual annotation as a reference for deviation. Count the number of items in the verification results that are inconsistent with the manual annotation. If the number of items after the supplementation is greater than that before the supplementation, it is determined that the deviation has increased, and the system is rolled back to the identity recognition rules, key fields or statistical design parameter registration database before the supplementation. If the deviation has not increased, the rules after the supplementation are retained.

[0015] In addition, the present invention also provides a project approval material consistency verification system based on field-level information extraction to realize the above-mentioned project approval material consistency verification method based on field-level information extraction, including: The document receiving module is used to receive multiple electronic documents of project approval materials to be verified, normalize the format of the electronic documents, and identify the type of each document based on the document content characteristics, so as to obtain a set of documents labeled with document types; the document types include proposals and statistical analysis plans. The rule creation module is used to set up material lists, identification rules, and key fields to be extracted according to document type, and to establish a statistical design parameter registration database. The field extraction module is used to extract field-level information from each document in the document collection according to key fields and field roles to obtain field values; based on identity recognition rules, it matches the identity information in the field values ​​of the subordinate documents with the identities of the documents present to obtain matching results; The checklist verification module is used to compare the document set with the material checklist to obtain the checklist verification results; The consistency comparison module is used to disambiguate field values ​​under the same field role in the same document to obtain representative values; based on the matching results, it generates document pairs for cross-file comparison, compares the representative values ​​of the same field role in the document pairs, and obtains cross-file consistency comparison results; The sample verification module is used to retrieve the back-inference method from the statistical design parameter registry based on the statistical design parameters in the field values, obtain the allowable range of the total number of cases enrolled, compare the total number of cases enrolled in the field values ​​with the allowable range, and obtain the sample size closure verification result. The report generation module is used to summarize the checklist verification results, cross-document consistency comparison results, and sample size closure verification results to generate consistency verification results.

[0016] The present invention has the following advantages over the prior art: (1) By setting material lists, identity recognition rules and key fields according to document types, field-level information extraction is performed on each document in the document set, and identity matching, list comparison, cross-document comparison and sample closure verification are carried out. Compared with the manual page-by-page comparison method, the verification efficiency and traceability of results in the multi-document association review scenario are improved.

[0017] (2) Matching results are obtained by matching the identity information in the field values ​​through identity recognition rules, and document pairs for cross-file comparison are generated based on the matching results. This enables the document association relationship to be accurately identified in cases such as version iteration and incomplete submission materials, avoiding misjudging normal version evolution as a consistency error, and also avoiding the problem of silent omission due to missing reference targets.

[0018] (3) By disambiguating the field values ​​of the same document and the same field role according to the field role, fields with similar expressions but different meanings will no longer be misjudged as self-contradictory. At the same time, by combining the numerical logical relationship between the representative values ​​for verification, it is possible to identify error types where the numerical values ​​are independent on the surface but the internal logical relationship is not valid, thus improving the accuracy of field-level verification.

[0019] (4) By retrieving the corresponding reverse calculation method from the statistical design parameter registration database according to the statistical design parameters, the allowable range of the total number of cases to be included is obtained and compared with the declared total number of cases to be included. This extends the sample size verification from the numerical surface comparison to the statistical design logic level, and can discover situations where the surface numerical values ​​are consistent but the statistical design logic is inconsistent, thus making up for the lack of verification capability of the general audit method at the statistical logic level. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the project initiation material consistency verification method based on field-level information extraction according to the present invention; Figure 2 This is a schematic diagram of the structure of the project initiation material consistency verification method based on field-level information extraction of the present invention; Figure 3 This is a schematic diagram of identity recognition and matching for the project initiation material consistency verification method based on field-level information extraction of the present invention. Figure 4 This is a sequence diagram of identity recognition matching for the project initiation material consistency verification method based on field-level information extraction, as presented in this invention. Detailed Implementation

[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 and Figure 2 As shown, this invention provides a method for verifying the consistency of project initiation materials based on field-level information extraction, including: S1. Receive multiple electronic documents of project approval materials to be verified, normalize the format of the electronic documents, and identify the type of each document based on the document content characteristics to obtain a set of documents labeled with document types; document types include proposals and statistical analysis plans.

[0024] Understandably, this invention receives multiple electronic documents submitted by sponsors or researchers through an upload interface. Document formats include, but are not limited to, PDF, Word, and scanned images. For scanned images and PDFs with unselectable text, optical character recognition (OCR) is first performed to obtain the text content. For Word documents and PDFs with selectable text, the text layer content is directly extracted. After text extraction, a unified format normalization process is performed, including removing duplicate text in headers and footers, standardizing full-width and half-width characters, standardizing date formats, and merging paragraphs that break across pages, ensuring a consistent text structure for documents from different sources and with different layouts in subsequent processing. After format normalization, each document is type-identified based on its content features. Specifically, document title keywords, chapter title keywords, and document-specific sentence structures are extracted as features and matched against a pre-configured document type feature library. The identification results distinguish at least five categories: project proposals, protocols, informed consent forms, statistical analysis plans, and ethical approvals. Each document is then labeled with its corresponding document type, resulting in a set of documents labeled with document types.

[0025] When the same document matches multiple document type features or does not match any document type features, the document is marked as type pending and transferred to manual relabeling. The results of manual relabeling are used to determine the final document type and are also recorded in the rule feedback channel after generating the consistency verification results, which is used to iteratively supplement the document type feature library.

[0026] S2. Set up material lists, identification rules, and key fields to be extracted according to document type, and establish a statistical design parameter registration database; The material list includes missing criteria and completeness criteria. The missing criteria are used to determine whether a document type is not submitted and whether it is recorded as missing from the list. The completeness criteria are used to determine whether the missing document type will cause the comparison between other documents to be blocked. The identity recognition rules stipulate that: documents in the document collection are divided into present documents and subordinate documents according to whether their version identity can be directly determined based on their own field values. The rules also specify the identification method for the present documents and the method for extracting the identity information of the declared document from the field values ​​of subordinate documents. The key fields are set in a three-level hierarchy: document type, field name, and field role. Field roles are used to identify semantically corresponding and comparable fields in different documents.

[0027] It is understandable that submissions must indicate that the document type is not allowed to be missing in this submission. Optional submissions indicate that verification can still be completed based on existing materials even if the document type is missing, but the default basis must be noted in the verification results. The integrity judgment condition is used to indicate whether the missing document type affects the cross-file comparison between other documents.

[0028] When the checklist verification results indicate that cross-document comparison is blocked, disambiguation and the summation of the total number of cases in the group and the number of cases in each group are still performed on the same document, but document pairs are not generated and cross-document comparisons are not performed. If the document type marked as blocked is a statistical analysis plan or protocol document, the reverse calculation method is not invoked, and the situation is recorded in the sample size closure verification results and marked as unverifiable. When the checklist verification results do not indicate that cross-document comparison is blocked, document pairs are generated according to the matching results, and cross-document comparisons and sample size closure verifications continue to be performed. The reverse calculation method refers to the calculation method that calculates the allowable range of the total number of cases in the group using the corresponding formula in the registry based on the extracted statistical design parameters.

[0029] Taking a proposal as an example, if its missing criteria are registered as mandatory submission and its completeness criteria are registered as missing, comparison with other documents will be blocked. If no proposal is submitted for review, it will be recorded as missing in the checklist verification results, and cross-document comparison will be blocked. Taking a statistical analysis plan as an example, if its missing criteria are registered as optional submission and its completeness criteria are registered as missing, comparison with other documents will not be blocked. That is, if no statistical analysis plan is submitted for review, the verification will not be considered to be suspended due to a missing checklist, but the statistical design parameters in the proposal will be used as the calculation basis in the sample size calculation stage.

[0030] Taking the proposal document as the primary document and the informed consent form as the subordinate document as an example, the identity recognition rules stipulate that the version number and version date of the proposal on which it is based are extracted from the field values ​​of the informed consent form as identity information. The extraction method is to locate typical sentences such as "This informed consent form is based on the version of the proposal" or "Based on the proposal version" in the informed consent form text, and parse the version number and date from the subsequent text of the sentence. At the same time, it is stipulated that the identity recognition method of the primary document is to directly read the version number and version date field values ​​marked on the proposal document itself. When the parsed version number has a non-numeric suffix or is not in a standardized format, making it impossible to compare the numerical values, the version number is downgraded and marked as a non-standard version number. At this time, the identity correspondence is determined only based on the order of the version date, and the verification result is noted as non-standard version number, judged by date order. If the version date cannot be uniquely determined, the identity recognition result of the document is marked as undeterminable and transferred to manual review, and it is not included in the automatic matching.

[0031] The identity of the proposal and informed consent form is recorded as the version pair ( ),in For version number, For version date, This refers to the document. Neither party, as the cited party, is required to declare any other documents on which it is based. The identity of the statistical analysis plan is also recorded as a version pair. The identity of an ethics approval document is recorded as the approval number and approval date. The identity of a project application is recorded as the instance submitted this time; version number and version date are allowed to be empty.

[0032] Identity matching is determined by exact match: the proposal, informed consent form, and statistical analysis plan must have the same version number and version date; the ethics approval document must have the same approval number; the project application is identified based on the example submitted this time. If the subordinate document declares that it is based on proposal version V2.0, and the proposal submitted for review is version V2.1, the two identities do not match, and the matching result is recorded as "declared but not present," which does not constitute a document pair for comparing field values.

[0033] The chronological order is only used within the same document series to determine version evolution, i.e., comparing proposals with each other, informed consent forms with each other, or statistical analysis plans with each other. It is not used for cross-series comparisons or for determining identity consistency between subordinate documents and documents present at the site. When two documents present at the site within the same series satisfy the chronological order but have different field values, the difference is categorized as a version evolution difference warning, not a field value inconsistency defect. If the parsed version number has a non-numeric suffix or is formatted incorrectly, making numerical comparison impossible, the version number is downgraded and marked as a non-standard version number. The evolution relationship within the series is then determined solely by the chronological order of the version dates, and the verification result is noted as non-standard version number, determined by date sequence. If the version date cannot be uniquely determined, the document's identity recognition result is marked as undeterminable and transferred to manual review, not participating in automatic matching.

[0034] In addition to the role-related fields for sample size, all document types have designated the primary endpoint, endpoint type, and study principal investigator as fields to be extracted. The endpoint type can be continuous, binary, or survival-related. The primary endpoint type has the same meaning as the endpoint type and is uniformly entered into the endpoint type after extraction. Protocol documents and statistical analysis plans additionally define the hypothesis type, allocation ratio, test direction, test method, dropout rate, significance level, and statistical power as fields to be extracted. For continuous endpoints, effect size and standard deviation are also extracted; for survival endpoints, control group event rate, enrollment period, and follow-up period are also extracted.

[0035] The hypothesis type can be superior, non-inferior, or equivalent; the test direction can be one-sided or two-sided; the test method is distinguished according to the endpoint type, with continuous endpoints including the two independent samples test and survival endpoints including the log-rank test. For binary endpoints, no executable backpropagation method is registered in this embodiment. When the endpoint type is binary, no corresponding backpropagation method exists in the statistical design parameter registration database, and the backpropagation method is not retrieved.

[0036] In one embodiment of the present invention, each entry in the statistical design parameter registry consists of four parts: a design tuple, a mandatory parameter table, a back-calculation method, and a rounding rule. The design tuple is composed of five values: endpoint type, hypothesis type, allocation ratio, test direction, and test method. The mandatory parameter table specifies the parameters that must be available before retrieving the back-calculation method: for continuous superiority, non-inferiority, or equivalent designs, the mandatory parameter is the significance level. Inspection efficiency Allocation ratio Effect size Standard deviation and shedding rate The essential parameter for survival-oriented design is the validation level. Inspection efficiency Allocation ratio Event rate, enrollment period, follow-up period, and dropout rate in the control group If any parameter in the required parameter table is missing, the statistical design parameters are considered incomplete, and the back-reasoning method will not be retrieved. The following two types of executable back-reasoning methods are pre-registered in the registry: The first type is for continuous endpoints, two independent samples, and superiority tests. The allocation ratio is noted. This represents the ratio of the number of cases in the experimental group to the number of cases in the control group. When the test is one-sided, the adjusted value of the significance level is recorded. Equal to the original inspection level When the test direction is bilateral, record equal Half of the values. If no one-sided or two-sided annotations are extracted from the field values, treat them as two-sided and indicate this in the verification results by default. Control group continuous solution Continuous solutions of the experimental group Calculate using the following formula, without rounding or scaling up based on the shedding rate: ; ; ; in , These represent the confidence levels corresponding to the standard normal distribution. , quantiles, For effect size, Standard deviation For the shedding rate, This is the floor function operator; This is a point estimate obtained by amplifying the dropout rate from the continuous solution. It is only used to illustrate the relationship with the dropout rate and is not directly used as the allowable range of values. For non-inferiority designs, the effect size in the above formula is replaced with the difference between the effect size and the non-inferiority threshold. For equivalent designs, the above formula is calculated by substituting the positive and negative two-sided thresholds respectively, and the larger of the required number of cases is taken as the back-calculation result.

[0037] The second type of design targets survival endpoints. The required number of events is given by the event probability formula under the assumption of an exponential distribution, based on the significance level, statistical power, and the event rate of the control group. The total number of cases to be enrolled is calculated by reverse formula: ; in The required number of events, This is the probability of an event occurring calculated under the assumption of an exponential distribution, based on the event rates of the enrollment period, follow-up period, and control group. For the shedding rate, This is the rounding up operator. The formula already includes amplification based on the dropout rate and rounding up; the result serves as the point value for the allowable range of the total number of cases included in a survival-based endpoint design. For continuous endpoint designs, the allowable range is not determined using this formula, but rather by the following two discretization paths. The registry also explicitly lists design categories that are not automatically calculated, including interim analysis-adjusted designs, adaptive designs, simulation-based sample size designs, multi-terminal designs, and clustered random designs. When a design tuple in a field falls into one of these categories, there is no corresponding back-calculation method in the registry, allowing step S6 to determine whether to retrieve the back-calculation method.

[0038] like Figure 3 and Figure 4 S3. For each document in the document collection, perform field-level information extraction according to key fields and field roles to obtain field values; based on identity recognition rules, match the identity information in the field values ​​of the subordinate documents with the identities of the documents present to obtain matching results.

[0039] In one embodiment of the present invention, a combination of rule template matching and a natural language processing model is used to extract field-level information from each document. The natural language processing model can be a sequence labeling model pre-trained based on the BERT structure and fine-tuned for text in this field, used for word segmentation, named entity recognition, and field boundary localization of the document text. The recognition results are merged and verified with the rule template extraction results. Candidate field values ​​not covered by the rule template but identified by the model are transferred to a low-confidence candidate set for manual review and confirmation before being included in the field values. During extraction, multiple sources are allowed for the same document and the same field role; that is, the value of a field role in the same document may appear in different chapters or paragraphs. The system iterates through all sources one by one and records the source coordinates for each extraction result, including the file name, page number, and paragraph position.

[0040] Specifically, the identity information retrieved from the subordinate document fields will be matched with the identity of the documents present, including: If no identity information is found in the field values, the matching result will be recorded as undeclared. When a field value contains identity information, and the document collection contains a document whose identity matches the identity information, the matching result is recorded as "declared and present". If a field value contains identity information, and there is no document in the document collection that matches the identity information, the matching result is recorded as "declared but not present".

[0041] In one embodiment of the present invention, the subordinate document declares that it is based on scheme version V2.0, while the actual scheme version in the submitted collection is V2.1. The two are inconsistent, that is, it is recorded as a declaration but not present.

[0042] Identity matching means that the identity information declared by the subordinate document and the identity of the document present are precisely matched according to the identity recognition rules. Figure 3 This indicates the determination process for the above three matching results. Among them, items that are declared but not present and are recorded in the cross-file consistency comparison results are considered identity mismatches, including version number mismatches, version date mismatches, or approval number mismatches, but not limited to version number mismatches. Figure 4 In the process, the verification engine first extracts the identity information on which the subordinate document is based according to the identity recognition rules, then reads the identity marked by the document itself, and outputs the results after comparing the two: not declared, declared and present, or declared but not present.

[0043] When the matching result is undeclared and there is only one document present in the document collection identified according to the identity recognition rules, the document with no identity information in the field value is paired with the document present. When there are zero or more documents present, no document pair is generated, and the situation is recorded in the cross-file consistency comparison result to avoid silent omission.

[0044] S4. Compare the document set with the material list to obtain the list verification results; Specifically, step S4 includes: Document types marked as mandatory in the materials list that do not exist in the document collection will be recorded as missing documents in the list verification results. Document types marked as optional submission in the material list and not present in the document set are not considered missing from the list, and the verification continues. The verification results will indicate that the type was not submitted and the default basis used. The default basis refers to other submitted document types or their field values ​​used in the verification when the document type was not submitted. Document types that are marked as missing in the material list and that prevent the comparison of other documents, and which do not exist in the document set, will be marked as preventing cross-document consistency comparison in the list verification results.

[0045] This invention uses identity recognition rules to match identity information in field values ​​to obtain matching results, and generates document pairs for cross-file comparison based on the matching results. This enables accurate identification of document relationships in situations such as version iteration and incomplete submission materials, avoiding misjudging normal version evolution as consistency errors, and also avoiding the problem of silent omissions due to missing reference targets.

[0046] S5. Disambiguate the field values ​​under the same field role in the same document to obtain representative values; generate document pairs for cross-file comparison based on the matching results, compare the representative values ​​of the same field role in the document pairs to obtain cross-file consistency comparison results; wherein, the statistical design parameters include endpoint type, hypothesis type, allocation ratio, test direction, test method and dropout rate.

[0047] This invention sets up material lists, identity recognition rules, and key fields according to document types, and performs field-level information extraction and identity matching, list comparison, cross-document comparison, and sample size closure verification for each document in the document set. Compared with the method of manual page-by-page comparison, it improves the verification efficiency and traceability of results in multi-document association review scenarios.

[0048] In one embodiment of the present invention, generating document pairs for cross-file comparison based on the matching results specifically includes: When the matching result is "declared and present", the document declaring the identity information is paired with the present document that matches the identity; if at least two documents are matched with the same present document as "declared and present", then these documents are further paired with each other to form document pairs. When the matching result is undeclared, and there is only one document present in the document collection identified according to the identity recognition rules, the document with no identity information in the field value is paired with the present document to form a document pair; If the matching result is a declaration but not present, no document pair is generated, and the matching result is recorded in the cross-file consistency comparison result.

[0049] In one embodiment of the present invention, the field roles include the total number of cases enrolled, the number of evaluable cases, the number of completed cases, and the number of cases in each group.

[0050] In one embodiment of the present invention, disambiguation of field values ​​under the same document and the same field role specifically includes: Field values ​​under the same document and the same field role are grouped into the value set of the field role, while field values ​​under different field roles are not grouped into the same value set; When all values ​​in the set are the same, the value is used as the representative value. When there are different values ​​in the value set, the field role does not output the representative value; Field roles whose values ​​have been output will be used to compare the same field roles in document pairs; field roles whose values ​​have not been output will not be compared in document pairs.

[0051] This invention disambiguates field values ​​within the same document and under the same field role by classifying them by field role. This prevents fields with similar expressions but different meanings from being misjudged as contradictory. At the same time, it combines the verification of numerical logical relationships between representative values ​​to identify error types where numerical values ​​appear independent but their internal logical relationships are invalid, thereby improving the accuracy of field-level verification.

[0052] In one embodiment of the present invention, comparing the representative values ​​of the same field roles in a document pair specifically includes: After obtaining the representative values, the representative values ​​of the total number of cases in the group are summed and compared with the representative values ​​of the number of cases in each group; If the sum of the number of cases in each group is not equal to the total number of cases in the group, it is determined that the relationship between roles is inconsistent, and the total number of cases in the document and the number of cases in each group are not included in the comparison of document pairs. The representative values ​​of the same field under the same role in the document pair that were not judged as having inconsistent relationships between roles are compared one by one. Inconsistent values ​​are recorded in the cross-file consistency comparison results.

[0053] In one embodiment of the present invention, the method for retrieving the back-calculation method from the statistical design parameter registry specifically includes: Based on the endpoint type, hypothesis type, allocation ratio, test direction, and test method in the field values, the corresponding reverse calculation method is determined and retrieved from the statistical design parameter registry. When the statistical design parameters of the statistical analysis plan are complete, reverse calculation is performed using the statistical design parameters of the statistical analysis plan and the total number of cases to be included in the field values ​​of the protocol document; If the proposed design and the statistical analysis plan give different values ​​for the same statistical design parameter, the difference will be recorded in the cross-document consistency comparison results; otherwise, the parameter in the statistical analysis plan will still be used. If there is no statistical analysis plan in the document collection and the statistical design parameters in the scheme document are complete, the reverse calculation is performed using the statistical design parameters in the scheme document; if the statistical design parameters are incomplete, or there is no corresponding reverse calculation method in the statistical design parameter registration database, the reverse calculation method is not invoked.

[0054] This invention retrieves the corresponding reverse calculation method from the statistical design parameter registration database according to the statistical design parameters, obtains the allowable range of the total number of cases to be included, and compares it with the declared total number of cases to be included. This extends the verification of sample size from the numerical surface comparison to the level of statistical design logic, and can discover situations where the surface numerical values ​​are consistent but the statistical design logic is inconsistent. This makes up for the lack of verification capability at the statistical logic level of general auditing methods.

[0055] In one embodiment of the present invention, the allowable range of values ​​for the total number of enrolled cases includes: The statistical design parameters were processed using a reverse calculation method to obtain a continuous solution that was not rounded down or amplified by the dropout rate. For continuous solutions, perform two separate operations: first round up and then amplify by the dropout rate, and second amplify by the dropout rate and then round up, to obtain two sets of integer results. The smaller of the two integer results is taken as the lower limit of the allowable range. The interval width is determined by the dropout rate, which is the difference between two divided by one and the dropout rate. The upper limit of the allowable range is the sum of the lower limit and the interval width minus one. The total number of enrolled cases is compared with the allowable range. If the total number of enrolled cases falls within the allowable range, it is recorded as a sample size closure check result and marked as closed. If it does not fall within the allowable range, it is recorded as a sample size closure check result and marked as a sample size design not closed.

[0056] Specifically, the formula for calculating the interval width is: ; in The interval width is the sum of the lower limit and the interval width minus one, which is used as the upper limit of the allowed value range. Therefore, the allowed value range is... .

[0057] A specific embodiment will be used for illustration: The statistical design parameters in the statistical analysis plan or proposal are processed using a reverse reasoning method to obtain the continuous solution for the experimental group. Continuous solutions compared to the control group Continuous solutions are not rounded down or amplified according to the dropout rate. It is the same quantity as the point estimate here, i.e. This is only used to illustrate the relationship with the shedding rate and is not directly used as the allowable range of values.

[0058] For continuous solutions, two discretization paths are executed: Path 1 first rounds up and then amplifies according to the dropout rate: ; Path two involves first scaling up the drop rate and then rounding it up: ; in, The total number of cases enrolled is obtained by first rounding down and then amplifying it according to the dropout rate. The total number of cases included is obtained by first scaling up the dropout rate and then rounding down. This refers to the shedding rate.

[0059] Pick , The smaller value in the range is used as the lower limit of the allowed value range. Determine the interval width ; in The interval width is the sum of the lower limit and the interval width minus one, which is used as the upper limit of the allowed value range. Therefore, the allowed value range is... .

[0060] Compare the total number of enrolled cases in the field with the allowed range of values: if the total number of enrolled cases falls within the allowed range, record it as a sample size closure check result and mark it as closed; if it does not fall within the allowed range, record it as a sample size closure check result and mark it as a sample size design not closed.

[0061] S6. Based on the statistical design parameters in the field values, retrieve the reverse calculation method from the statistical design parameter registry to obtain the allowable range of the total number of cases enrolled. Compare the total number of cases enrolled in the field values ​​with the allowable range to obtain the sample size closure verification result. S7. Summarize the list verification results, cross-document consistency comparison results, and sample size closure verification results to generate consistency verification results.

[0062] Specifically, step S7 includes: The results of the checklist verification, cross-document consistency comparison, and sample size closure verification are categorized by type. The classification results are sorted according to a preset priority, and a verification text is generated to obtain the consistency verification result. The manual annotations are used to confirm, identify false alarms or correct errors in the consistency verification results. Based on these manual annotations, the database of identity recognition rules, key fields or statistical design parameters is supplemented. Re-execute field-level information extraction and comparison on the document collection, compare the verification results before and after the supplementation, and use manual annotation as a reference for deviation. Count the number of items in the verification results that are inconsistent with the manual annotation. If the number of items after the supplementation is greater than that before the supplementation, it is determined that the deviation has increased, and the system is rolled back to the identity recognition rules, key fields or statistical design parameter registration database before the supplementation. If the deviation has not increased, the rules after the supplementation are retained.

[0063] Understandably, "confirmation" indicates that the record is recognized as a genuine anomaly and is retained as a basis for communication and rectification with the applicant; "false alarm" indicates that the record is judged as a system error, and is marked as a false alarm and included in the error correction sample; "correction" indicates that the incorrect field values, field roles, or identity information are corrected and the manually annotated values ​​are used as the standard to write back to the verification result.

[0064] In one embodiment of the present invention, nine categories are preset: Category 1 is the absence of the referenced target version or contradictory version control information; Category 2 is missing list; Category 3 is mismatch between endpoint and statistical method; Category 4 is internal contradiction; Category 5 is inconsistency of fields across files; Category 6 is unclosed sample size design; Category 7 is undeclared reference, including prompts indicating that non-declared pairings have been performed; Category 8 is unverifiable, due to insufficient corresponding parameters or the registry not being covered; Category 9 is version evolution difference, corresponding to field differences caused by legitimate evolution within the same series, which is defaulted to a prompt rather than a defect. Category 8 is strictly separated from Category 6. Cases with insufficient parameters or statistical design tuples falling outside the coverage of the registry are all classified into Category 8 and should not be reclassified as unclosed due to failure to calculate residuals. Multiple records of different categories are allowed to exist simultaneously in the same file and for the same field, and other records are not discarded because a higher priority category is hit.

[0065] Furthermore, the priority order is the numbering order of the nine categories. Sort by filename or submission timestamp. For comparison results generated from non-declared pairings, regardless of whether inconsistencies are found, the non-declared pairing marker is retained in category seven. Pre-set text templates are provided for each category. The template for citing an absent target version is: File [Subordinate Filename] Page {Page Number} declares the basis [Type] Version {Version Number} ({Version Date}), and no file matching the identity exists in the submitted materials; for a present file of the same type, it is: [Present Filename] Version {Present Version Number} ({Present Date}). The template for an unclosed sample size design is: File [Declared Filename] Page {Page Number} records the total number of cases included in the group as { For example, the significance level is given on page {} of the [Statistical Analysis Plan File Name or Proposal File Name]. Inspection efficiency Allocation ratio Effect size Standard deviation shedding rate The range of the total number of cases in the reverse-engineered group is { }to{ For example, a declared value exceeds a range, where... Templates that cannot be verified include: the file [filename] statistical design is identified as {design tuple or external reason}, the statistical design parameter registration library has no corresponding executable item or the required parameter {parameter name} is missing, residuals are not calculated, and it is not judged as unclosed. Substitute the field values ​​of each record into the corresponding template to generate verification text. A structured data table is generated synchronously, with one record per row. Columns include exception category, priority, involved files, field name and role, value, coordinates, pairing method, and generated text description. It supports filtering by filename, field name, and exception category, and can be exported to common office document formats.

[0066] This invention also provides a project approval material consistency verification system based on field-level information extraction, used to implement the project approval material consistency verification method based on field-level information extraction as described above, including: The document receiving module is used to receive multiple electronic documents of project approval materials to be verified, normalize the format of the electronic documents, and identify the type of each document based on the document content characteristics, so as to obtain a set of documents labeled with document types; the document types include proposals and statistical analysis plans. The rule creation module is used to set up the bill of materials, identification rules and key fields to be extracted according to document type, and to establish a statistical design parameter registration database. The field extraction module is used to extract field-level information from each document in the document collection according to key fields and field roles to obtain field values; based on identity recognition rules, it matches the identity information in the field values ​​of the subordinate documents with the identities of the documents present to obtain matching results; The checklist verification module is used to compare the document set with the material checklist to obtain the checklist verification results; The consistency comparison module is used to disambiguate field values ​​under the same field role in the same document to obtain representative values; based on the matching results, it generates document pairs for cross-file comparison, compares the representative values ​​of the same field role in the document pairs, and obtains cross-file consistency comparison results; The sample verification module is used to retrieve the back-inference method from the statistical design parameter registry based on the statistical design parameters in the field values, obtain the allowable range of the total number of cases enrolled, compare the total number of cases enrolled in the field values ​​with the allowable range, and obtain the sample size closure verification result. The report generation module is used to summarize the checklist verification results, cross-document consistency comparison results, and sample size closure verification results to generate consistency verification results.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A project material consistency verification method based on field-level information extraction, characterized in that, include: S1. Receive multiple electronic documents of project approval materials to be verified, normalize the format of the electronic documents, and identify the type of each document based on the document content characteristics to obtain a set of documents labeled with document types; document types include proposals and statistical analysis plans. S2. Set up material lists, identification rules, and key fields to be extracted according to document type, and establish a statistical design parameter registration database; The material list includes missing criteria and completeness criteria. The missing criteria are used to determine whether a document type is not submitted and whether it is recorded as missing from the list. The completeness criteria are used to determine whether the missing document type will cause the comparison between other documents to be blocked. The identity recognition rules stipulate that: documents in the document collection are divided into present documents and subordinate documents according to whether their version identity can be directly determined based on their own field values. The rules also specify the identification method for the present documents and the method for extracting the identity information of the declared document from the field values ​​of subordinate documents. The key fields are set in a three-level hierarchy: document type, field name, and field role. Field roles are used to identify semantically corresponding and comparable fields in different documents. S3. For each document in the document collection, perform field-level information extraction according to key fields and field roles to obtain field values; based on identity recognition rules, match the identity information in the field values ​​of the subordinate documents with the identities of the documents present to obtain matching results; S4. Compare the document set with the material list to obtain the list verification results; S5. Disambiguate the field values ​​under the same field role in the same document to obtain representative values; generate document pairs for cross-file comparison based on the matching results, compare the representative values ​​of the same field role in the document pairs to obtain cross-file consistency comparison results; S6. Based on the statistical design parameters in the field values, retrieve the reverse calculation method from the statistical design parameter registry to obtain the allowable range of the total number of cases enrolled. Compare the total number of cases enrolled in the field values ​​with the allowable range to obtain the sample size closure verification result. S7. Summarize the list verification results, cross-document consistency comparison results, and sample size closure verification results to generate consistency verification results.

2. The project material consistency verification method based on field-level information extraction according to claim 1, characterized in that: Step S4 specifically includes: Document types marked as mandatory in the materials list that do not exist in the document collection will be recorded as missing documents in the list verification results. Document types marked as optional submission in the material list and not present in the document set are not considered missing from the list, and the verification continues. The verification results will indicate that the type was not submitted and the default basis used. The default basis refers to other submitted document types or their field values ​​used in the verification when the document type was not submitted. Document types that are marked as missing in the material list and that prevent the comparison of other documents, and which do not exist in the document set, will be marked as preventing cross-document consistency comparison in the list verification results. 3.The project material consistency verification method based on field-level information extraction according to claim 1, characterized in that: The process of matching the identity information retrieved from the subordinate document fields with the identity of the documents present specifically includes: If no identity information is found in the field values, the matching result will be recorded as undeclared. When a field value contains identity information, and the document collection contains a document whose identity matches the identity information, the matching result is recorded as "declared and present". If a field value contains identity information, and there is no document in the document collection that matches the identity information, the matching result is recorded as "declared but not present".

4. The project material consistency verification method based on field-level information extraction according to claim 3, characterized in that: The generation of document pairs for cross-file comparison based on the matching results specifically includes: When the matching result is "declared and present", the document declaring the identity information is paired with the present document that matches the identity; if at least two documents are matched with the same present document as "declared and present", then these documents are further paired with each other to form document pairs. When the matching result is undeclared, and there is only one document present in the document collection identified according to the identity recognition rules, the document with no identity information in the field value is paired with the present document to form a document pair; If the matching result is a declaration but not present, no document pair is generated, and the matching result is recorded in the cross-file consistency comparison result.

5. The project material consistency verification method based on field-level information extraction according to claim 1, characterized in that: The field roles include the total number of cases enrolled, the number of evaluable cases, the number of completed cases, and the number of cases in each group. The disambiguation of field values ​​under the same document and the same field role specifically includes: Group the field values ​​under the same document and the same field role into the value set of the field role; When all values ​​in the set are the same, the value is used as the representative value. When there are different values ​​in the value set, the field role does not output the representative value; Field roles whose values ​​have been output will be used to compare the same field roles in document pairs; field roles whose values ​​have not been output will not be compared in document pairs.

6. The project material consistency verification method based on field-level information extraction according to claim 5, characterized in that: The comparison of representative values ​​for the same field role in the document pair specifically includes: After obtaining the representative values, the representative values ​​of the total number of cases in the group are summed and compared with the representative values ​​of the number of cases in each group; If the sum of the number of cases in each group is not equal to the total number of cases in the group, it is determined that the relationship between roles is inconsistent, and the total number of cases in the document and the number of cases in each group are not included in the comparison of document pairs. The representative values ​​of the same field under the same role in the document pair that were not judged as having inconsistent relationships between roles are compared one by one. Inconsistent values ​​are recorded in the cross-file consistency comparison results.

7. The project material consistency verification method based on field-level information extraction according to claim 1, characterized in that: The statistical design parameters include endpoint type, hypothesis type, allocation ratio, test direction, test method, and dropout rate. The method for retrieving the back-calculation method from the statistical design parameter registry specifically includes: Based on the endpoint type, hypothesis type, allocation ratio, test direction, and test method in the field values, the corresponding reverse calculation method is determined and retrieved from the statistical design parameter registry. When the statistical design parameters of the statistical analysis plan are complete, reverse calculation is performed using the statistical design parameters of the statistical analysis plan and the total number of cases to be included in the field values ​​of the protocol document; If the proposed design and the statistical analysis plan give different values ​​for the same statistical design parameter, the difference will be recorded in the cross-document consistency comparison results; otherwise, the parameter in the statistical analysis plan will still be used. If there is no statistical analysis plan in the document collection and the statistical design parameters in the scheme document are complete, the reverse calculation is performed using the statistical design parameters in the scheme document; if the statistical design parameters are incomplete, or there is no corresponding reverse calculation method in the statistical design parameter registration database, the reverse calculation method is not invoked.

8. The project material consistency verification method based on field-level information extraction according to claim 7, characterized in that: The allowed range of values ​​for obtaining the total number of enrolled cases includes: The statistical design parameters were processed using a reverse calculation method to obtain a continuous solution that was not rounded down or amplified by the dropout rate. For continuous solutions, perform two separate operations: first round up and then amplify by the dropout rate, and second amplify by the dropout rate and then round up, to obtain two sets of integer results. The smaller of the two integer results is taken as the lower limit of the allowable range. The interval width is determined by the dropout rate, which is the difference between two divided by one and the dropout rate. The upper limit of the allowable range is the sum of the lower limit and the interval width minus one. The total number of enrolled cases is compared with the allowable range. If the total number of enrolled cases falls within the allowable range, it is recorded as a sample size closure check result and marked as closed. If it does not fall within the allowable range, it is recorded as a sample size closure check result and marked as a sample size design not closed. 9.The project material consistency verification method based on field-level information extraction of claim 1, wherein: Step S7 specifically includes: The results of the checklist verification, cross-document consistency comparison, and sample size closure verification are categorized by type. The classification results are sorted according to a preset priority, and a verification text is generated to obtain the consistency verification result. The manual annotations are used to confirm, identify false alarms or correct errors in the consistency verification results. Based on these manual annotations, the database of identity recognition rules, key fields or statistical design parameters is supplemented. Re-execute field-level information extraction and comparison on the document collection, compare the verification results before and after the supplementation, and use manual annotation as a reference for deviation. Count the number of items in the verification results that are inconsistent with the manual annotation. If the number of items after the supplementation is greater than that before the supplementation, it is determined that the deviation has increased, and the system is rolled back to the identity recognition rules, key fields or statistical design parameter registration database before the supplementation. If the deviation has not increased, the rules after the supplementation are retained.

10. A project material consistency verification system based on field-level information extraction, characterized in that, The system is used to implement the project initiation material consistency verification method based on field-level information extraction as described in any one of claims 1-9, including: The document receiving module is used to receive multiple electronic documents of project approval materials to be verified, normalize the format of the electronic documents, and identify the type of each document based on the document content characteristics, so as to obtain a set of documents labeled with document types; the document types include proposals and statistical analysis plans. The rule creation module is used to set up material lists, identification rules, and key fields to be extracted according to document type, and to establish a statistical design parameter registration database. The field extraction module is used to extract field-level information from each document in the document collection according to key fields and field roles to obtain field values; based on identity recognition rules, it matches the identity information in the field values ​​of the subordinate documents with the identities of the documents present to obtain matching results; The checklist verification module is used to compare the document set with the material checklist to obtain the checklist verification results; The consistency comparison module is configured to disambiguate the field values under the same document and the same field role to obtain a representative value; generate a document pair for cross-file comparison based on the matching result, compare the representative values of the same field role in the document pair, and obtain a cross-file consistency comparison result; The sample verification module is configured to obtain a reverse deduction method from a statistical design parameter registration library based on a statistical design parameter in the field value, obtain an allowed value range of the total number of enrolled cases, compare the total number of enrolled cases in the field value with the allowed value range, and obtain a sample size closure verification result; The report generation module is configured to summarize the list verification result, the cross-file consistency comparison result, and the sample size closure verification result to generate a consistency verification result.

Images