Methods and systems for verifying the consistency of review results using artificial intelligence
By generating a structured review dataset, correcting rating scale bias, identifying outlier experts, and performing dynamic threshold verification, the consistency problem of review results in the review system was solved, and the accuracy and consistency of review results were verified.
Patent Information
- Application Number
- CN202611357626.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-03
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, review systems struggle to effectively handle expert scoring scale biases and outlier scores, making it difficult to identify substantial differences between AI review results and expert review results. Furthermore, cross-batch review data is difficult to aggregate, affecting the consistency and accuracy of review results.
By reading review form data, a structured review dataset is generated, a baseline snapshot is bound, independent inference is performed, scoring scale offset is corrected, outlier experts are identified, robust expert mean is calculated, dynamic deviation threshold is determined, and cross-batch mapping is performed through indicator hierarchy constraints and batch mutual exclusion constraints, scoring components are fused, and a consistency verification report is generated.
It reduces the impact of scoring scale bias and outlier scores on consistency verification, improves the accuracy and consistency of review results, ensures the effective collection of historical data and dynamic updating of judgment parameters, and reduces false positive and false negative rates.
Smart Images

Figure CN123118049A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, specifically to a method and system for verifying the consistency of review results in artificial intelligence. Background Technology
[0002] In project application review, professional title evaluation, and scientific and technological achievement evaluation, the application review system is used to manage application materials, review indicators, review processes, and review results. Review experts submit sub-item scores, sub-item comments, and final review conclusions according to the configured review indicators. Some systems introduce review analysis language models to generate AI review results, and use manual comparison or fixed threshold comparison to check the differences between AI review results and expert review results.
[0003] Different experts have varying degrees of understanding of the scoring scale, with some experts giving overall scores that are either too high or too low across multiple review indicators. This bias in the scoring scale coexists with the differences in scores for specific indicators within the scoring data. Directly comparing the average expert scores with a fixed threshold is susceptible to the influence of scoring scale bias, outlier scores, and differences in the distribution of indicator data.
[0004] The review indicator system is adjusted with each review batch, and the same logical indicator may have different indicator names, descriptions, or identifiers in different batches. The existing system usually stores data by batch indicator identifier, making it difficult to aggregate cross-batch review data to the same logical indicator, which limits the accumulation of effective historical samples and the statistical calculation of judgment parameters.
[0005] The decoding process of the language model used in the review analysis is random; multiple inferences based on the same application materials and review indicators will yield different results. Comparing results from a single inference attempt makes it difficult to measure the fluctuations in the AI output. Fixed judgment parameters cannot be updated based on the results of the review and confirmation, making it difficult to ensure that the judgment parameters match the historical data distribution and the business requirements for false positive and false negative rates. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides a method and system for verifying the consistency of AI review results, solving the problem that existing technologies lack processing for expert scoring scale deviations and outlier scores, which affects the identification of substantial differences between AI review results and expert review results.
[0007] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of this invention provides a method for verifying the consistency of review results using artificial intelligence, comprising the following steps: Read the review form data, bind the baseline snapshot and generate a structured review dataset, obtain the expert rating components, perform independent inference, and obtain the AI output dispersion and AI rating components; Read the indicator nodes for this batch from the structured review dataset, map them to the unified logical indicator nodes, and aggregate the expert rating components and AI rating components into the unified logical indicator nodes. The scoring scale offset is determined based on the expert's scoring components. The scoring scale is then corrected for experts who meet the preset correction conditions based on the scoring scale offset, resulting in the corrected scoring components. Outlier experts are identified based on the corrected scoring components, and the mean of robust experts is calculated. The AI expert index deviation is obtained based on the AI scoring components and the mean of robust experts. The dynamic deviation threshold used in this verification is obtained from the baseline snapshot. The actual deviation judgment threshold is obtained by combining the AI output dispersion. The difference index is marked based on the AI expert index deviation and the actual deviation judgment threshold, and the marked items are obtained. Attribution rules are applied to the marked items to obtain the main attribution label and the additional attribution label; a structured consistency verification report and a review work order are generated based on the main attribution label and the additional attribution label; the review confirmation result of the review work order is received; and the next baseline snapshot is generated based on the review confirmation result.
[0008] Furthermore, using the node deviation formula, the deviation sequence for this batch is constructed based on the median of each expert's score relative to the other experts, yielding the intra-batch offset and the proportion of directional consistency within this batch. The scoring scale profile is then read, and the historical offset and the proportion of historical directional consistency are obtained from the valid historical deviation sequence within the scoring scale profile. When both the intra-batch offset and the historical offset are valid, the scoring scale offset is determined using the scale offset formula, and the proportion of directional consistency is obtained based on the proportion of directional consistency within this batch and the proportion of historical directional consistency.
[0009] When the historical offset is empty, the ratio of the scoring scale offset to the deviation direction consistency is taken as the ratio of the batch offset to the current batch direction consistency. When the current batch deviation sequence length is less than the preset minimum deviation sequence length, the ratio of the scoring scale offset to the deviation direction consistency is taken as the ratio of the historical offset to the historical direction consistency. When the absolute value of the scoring scale offset is greater than the preset scale offset limit and the deviation direction consistency ratio is greater than the preset direction consistency ratio limit, the scoring scale offset is subtracted from the expert's scoring component. When the absolute value of the scoring scale offset is greater than the preset overcorrection upper limit, if the scoring scale offset is positive, the overcorrection upper limit is subtracted from the expert's scoring component; if it is negative, the overcorrection upper limit is added to the expert's scoring component to obtain the corrected scoring component.
[0010] Furthermore, the expert ratings aggregated to the unified logical indicator node are statistically analyzed to obtain the number of effective experts. The consistency distance is calculated by the absolute value of the difference between the corrected ratings and the median of the corrected ratings of the remaining experts. The sum of the median of all consistency distances and the absolute deviation of twice the median is used as the outlier threshold. The smaller of the number of effective experts divided by 3 and rounded down, and the number of effective experts minus the preset minimum number of effective experts, is taken as the upper limit for the number of experts to be removed. Experts whose consistency distance is greater than the outlier threshold are removed according to the upper limit. The arithmetic mean of the corrected ratings of the remaining experts is used as the robust expert mean.
[0011] When the baseline snapshot has corresponding historical parameters, the dynamic deviation threshold used in this verification is calculated using the dynamic threshold formula; otherwise, the dynamic deviation threshold used in this verification is the preset initial deviation threshold. When the AI output dispersion is empty, the actual deviation judgment threshold is the dynamic deviation threshold used in this verification; when the absolute value of the AI expert indicator deviation is greater than the actual deviation judgment threshold, it is marked as a difference indicator.
[0012] Furthermore, the indicator hierarchy of the indicator nodes in this batch is read from the structured review dataset, and a similarity matrix is generated under the constraint of the indicator hierarchy to obtain the similarity. Within the same batch, an indicator node in this batch can be mapped to at most one unified logical indicator node, and a unified logical indicator node can accept at most one indicator node in this batch. When the similarity is not less than the preset mapping similarity threshold, the mapping result is determined; when it is not less than the preset lower limit of the mapping gray area and less than the mapping similarity threshold, a mapping confirmation item is generated, and a temporarily added unified logical indicator node is obtained; when it is less than the lower limit of the mapping gray area, it is determined to be a newly added unified logical indicator node.
[0013] Temporarily added unified logical indicator nodes and newly added unified logical indicator nodes adopt an initial deviation threshold. The valid score components (excluding invalid scores) from expert ratings and AI ratings, along with the number of valid experts, are aggregated into the corresponding unified logical indicator nodes. The number of valid historical samples and the average of historical score components are obtained from the baseline snapshot. If the number of valid historical samples is not less than the preset minimum sample size, the similarity is not less than the mapping similarity limit, and the absolute value of the difference between the averages of historical score components is not greater than the scale offset limit, a node merging item is generated. After confirmation, the valid historical samples are included in the confirmed merged unified logical indicator nodes.
[0014] Furthermore, for the same application project and the same indicator node in this batch, independent inference is performed according to the preset number of AI samplings. The same prompt words and the same decoding parameters except for the random state are used in each inference. Valid sub-item scores are retained for each inference result. The median of the valid sub-item scores is taken as the AI sub-item score. The AI sub-item scores are converted into AI score components. The AI output dispersion is obtained from the score components of the valid sub-item scores. When the number of valid sub-item scores is less than 2, the AI output dispersion is empty.
[0015] When the AI output dispersion is not empty, the actual threshold formula is used. The actual deviation judgment threshold is calculated based on the dynamic deviation threshold used in this verification, the preset AI instability relaxation coefficient, and the AI output dispersion. When the AI output dispersion is greater than the preset AI stability limit, it is marked as an AI output instability indicator, an AI re-review task is generated, and the corresponding data is not included in the AI expert indicator deviation samples in the valid historical sample set.
[0016] Furthermore, a structured consistency verification report is generated based on the primary attribution label, additional attribution labels, and the tagged items. A review priority score is obtained based on the tagged items, and review work orders are generated according to the review priority score. Data that has not yet been tagged is grouped into untagged items. A first set of sampled review work orders and a second set of sampled review work orders are drawn from the tagged items and untagged items respectively according to a preset sampling review ratio. Review confirmation results are received, and the set of valid historical samples is updated accordingly.
[0017] When the number of valid historical samples is not less than the minimum sample size, the dynamic deviation threshold is recalculated using the dynamic threshold formula and written to the next baseline snapshot in the pending release state. The decision confidence multiple is read from the baseline snapshot, and the measured false positive rate is obtained from the first group of sampling review work orders, while the measured false negative rate is obtained from the second group of sampling review work orders and unmarked items.
[0018] If the measured false positive rate is greater than the preset target false positive rate but the measured false negative rate is not greater than the preset target false negative rate, the confidence factor is increased by the preset adjustment step size; if the measured false negative rate is greater than the target false negative rate but the measured false positive rate is not greater than the target false positive rate, the confidence factor is decreased by the adjustment step size; if the measured false positive rate is greater than the target false positive rate and the measured false negative rate is greater than the target false negative rate, an insufficient verification dimension prompt is generated and written to the structured consistency verification report. After this batch meets the preset termination conditions, the next baseline snapshot is switched from the pending release state to the released state, without rewriting the baseline snapshots already bound to this batch.
[0019] Furthermore, attribution rules are executed in the order of differences in scoring scale, AI applicability, indicator understanding, and unclassified differences, with the first result to be identified as the primary attribution label and the rest as secondary attribution labels.
[0020] A second aspect of the present invention provides a consistency verification system for review results of artificial intelligence, used to perform the aforementioned consistency verification method for review results of artificial intelligence, comprising: The review data acquisition module is used to read review form data, bind baseline snapshots and generate structured review datasets, obtain expert rating components, perform independent inference, and obtain AI output dispersion and AI rating components. The indicator semantic alignment module is used to read the indicator nodes of this batch from the structured review dataset, map them to the unified logical indicator nodes, and aggregate the expert rating components and AI rating components into the unified logical indicator nodes. The rating scale correction module is used to determine the rating scale offset based on the expert's rating components, and to perform rating scale correction on experts who meet the preset correction conditions based on the rating scale offset to obtain the corrected rating components. The consistency verification module is used to identify outlier experts and calculate the mean of robust experts based on the corrected score components. It obtains the AI expert index deviation based on the AI score components and the mean of robust experts. It obtains the dynamic deviation threshold used in this verification from the baseline snapshot. It obtains the actual deviation judgment threshold by combining the AI output dispersion. It marks the difference index based on the AI expert index deviation and the actual deviation judgment threshold to obtain the marked items. The conflict attribution analysis module is used to execute attribution rules on the marked items and obtain the main attribution label and the additional attribution label; The verification result generation module is used to generate a structured consistency verification report and a review work order based on the main attribution label and the additional attribution label, receive the review confirmation result of the review work order, and generate the next baseline snapshot based on the review confirmation result.
[0021] This invention provides a method and system for consistency verification of review results using artificial intelligence. It has the following beneficial effects: 1. This invention calculates node deviation using the leave-one-out method, integrates intra-batch offset with historical offset, and performs scoring scale correction based on the deviation direction consistency ratio. On this basis, it identifies outlier experts, calculates the mean of robust experts and dynamic deviation threshold, and reduces the impact of scoring scale deviation and outlier scores on consistency verification.
[0022] 2. This invention determines cross-batch mapping relationships through indicator hierarchy constraints and mutual exclusion constraints within the same batch. Combined with mapping confirmation items and node merging items, it aggregates the effective scoring components of different review batches into a unified logical indicator node, reducing the dispersion of historical data and providing effective historical samples for calculating judgment parameters by node.
[0023] 3. This invention calculates the AI output dispersion by repeated sampling and determines the actual deviation judgment threshold, incorporating AI output fluctuations into the difference judgment; updates the judgment parameters based on the sampling review results and publishes the next baseline snapshot, so that subsequent review batches adopt the updated results, while maintaining the consistency of the judgment parameters in this batch. Attached Figure Description
[0024] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention; Figure 3 This is a flowchart of the review data collection and preprocessing process for this invention; Figure 4 This is a flowchart of the cross-batch mapping of indicator nodes and the generation of review vectors in this invention; Figure 5 This is a flowchart of the scoring scale offset separation and deduction process of the present invention; Figure 6 This is a flowchart of the consistency verification process of the present invention; Figure 7 This is a flowchart of the conflict attribution classification process of the present invention; Figure 8 This is a flowchart illustrating the verification report generation, review feedback, and baseline snapshot update process of the present invention. Figure 9 This is a bar chart comparing the experimental effects of this embodiment and the comparative method in the application examples of the present invention. Detailed Implementation
[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] Please see the appendix Figure 1 This invention provides an artificial intelligence-based consistency verification system for review results, which is set in the application review system and includes: The review data acquisition module is used to generate structured review datasets, AI output dispersion and application material fragment indexes, and bind baseline snapshots.
[0027] The indicator semantic alignment module is used to determine the mapping relationship between the indicator nodes of this batch and the unified logical indicator nodes, collect the effective scoring components, and generate the review vector.
[0028] The scoring scale correction module is used to determine the scoring scale offset according to the node deviation formula and the scale offset formula, and generate the corrected scoring components.
[0029] The consistency verification module is used to verify the review results according to the dynamic threshold formula, the actual threshold formula, and the discrete baseline formula, and to obtain the marked items.
[0030] The conflict attribution analysis module is used to determine the primary and secondary attribution labels for the tagged items.
[0031] The verification result generation module is used to generate structured consistency verification reports and review work orders, maintain a set of valid historical samples based on the review confirmation results, and publish the next baseline snapshot.
[0032] See appendix Figure 2 This invention provides a method for verifying the consistency of review results using artificial intelligence, comprising the following steps: S1, the review data collection module collects review data, generates scoring components, AI output dispersion and application material fragment index, and binds baseline snapshots.
[0033] S2, the indicator semantic alignment module determines the mapping relationship between indicator nodes and unified logical indicator nodes, collects effective scoring components, and generates review vectors.
[0034] S3, the scoring scale correction module determines the scoring scale offset according to the node deviation formula and the scale offset formula, and generates the corrected scoring components.
[0035] S4, the consistency verification module performs consistency verification according to the dynamic threshold formula, the actual threshold formula and the discrete baseline formula, and obtains the marked items.
[0036] S5, the conflict attribution analysis module determines the primary attribution label and the secondary attribution label based on the tagged items.
[0037] S6, the verification result generation module generates a structured consistency verification report and a review work order, maintains a valid historical sample set based on the review confirmation results, and publishes the next baseline snapshot.
[0038] The technical solutions in the embodiments of the present invention will be further explained below in conjunction with the specific processing procedures.
[0039] In this embodiment, the application review system is equipped with a review analysis language model and a semantic encoding model, stores and maintains the configuration of review indicators, review forms, review subjects, review processes and models, and stores review batch information, application project information and application materials.
[0040] The review indicator configuration includes indicator name, indicator description, review indicator scoring data type, upper limit of scoring interval, lower limit of scoring interval, indicator level, indicator weight, parent review indicator identifier, indicator activation status, scoring rules, unit of measurement, numerical precision, statistical period format, object name, upper limit of total score, lower limit of total score, and passing score. The review form configuration records the correspondence between review form fields and review indicator identifiers. The review subject configuration includes review subject identifier, review subject role, and expert weight. The model configuration includes review analysis language model identifier, semantic encoding model identifier, prompt word version identifier, prompt words, prompt word template, and decoding parameters.
[0041] The application review system generates review process instances based on review batch information, application project information, review entity configuration, and review process configuration. It also receives and stores review form data through these review process instances. The review form data includes sub-item scores, sub-item comments, final review conclusions, final review opinions, and the role of the final review submitter.
[0042] See appendix Figure 3 In this embodiment, step S1 is executed through the review data acquisition module and includes the following sub-steps: S1.1 The review data acquisition module reads review form data from the review process instance of the application review system, reads the corresponding application materials according to the application project identifier, and reads the review indicator configuration, review form configuration, review subject configuration and model configuration associated with the review batch identifier from the application review system, and establishes associations according to the application project identifier, review batch identifier, review subject identifier and review indicator identifier.
[0043] Based on the correspondence between the review form fields and review indicator identifiers recorded in the review form configuration, the review indicator identifiers corresponding to the sub-item scores and sub-item comments are determined. The sub-item scores, sub-item comments, final review conclusions, final review opinions, and final review submitter roles are written into the structured review dataset. The indicator name, indicator description, review indicator score data type, upper limit of score interval, lower limit of score interval, indicator level, indicator weight, parent review indicator identifier, indicator activation status, scoring rules, unit of measurement, numerical precision, statistical period format, object name, review subject role, expert weight, review analysis language model identifier, semantic coding model identifier, and prompt word version identifier are read from the review indicator configuration, review subject configuration, and model configuration and written into the structured review dataset.
[0044] Identify the experts participating in the scoring based on their review subject roles, and associate the review subject identifier and expert weight of each expert with the corresponding scoring component.
[0045] The primary and secondary indicators for this review batch are determined based on the indicator hierarchy, and the primary indicator to which the secondary indicator belongs is determined based on the parent review indicator identifier of the secondary indicator. In this embodiment, the primary indicator is used to organize the secondary indicators and configure the weight of the primary indicator, and the secondary indicator is the scoreable indicator corresponding to the sub-item score and sub-item comment.
[0046] When the data type of the review indicator scores is numerical, the score component is obtained by subtracting the lower limit of the score interval from the original sub-score and then dividing by the difference between the upper and lower limits of the score interval. When the data type of the review indicator scores is graded, the grades are arranged from lowest to highest, with the lowest grade recorded as 0 and the highest grade as 1. The remaining grades are equidistant within the closed interval of 0 to 1. The pass and fail types correspond to 1 and 0, respectively. Scores with missing content, unrecognizable data type of review indicator scores, fewer than 2 grades, or an upper limit equal to the lower limit of the score interval are considered invalid. When the converted result exceeds the closed interval of 0 to 1, it is truncated to the corresponding endpoint, and the score component and the corresponding data validity identifier are written into the structured review dataset.
[0047] Multiply the weight of each secondary indicator under its primary indicator by the weight of that primary indicator to obtain the weight product of the corresponding secondary indicator; divide each weight product by the sum of the weight products of all secondary indicators that are in use and have valid weight configurations in this review batch to obtain the global weight of each secondary indicator.
[0048] If an indicator weight is missing, has an incorrect data type, or is less than 0, the corresponding indicator weight configuration is deemed invalid. If the sum of the weights of all secondary indicators is equal to 0, no global weight is generated, and an abnormality in the scoring rule configuration is generated.
[0049] Based on the read review indicator configuration, determine the lower and upper limits of the total score, and obtain the passing score associated with this review batch. Subtract the lower limit of the total score from the passing score, and then divide by the difference between the upper and lower limits of the total score to obtain the normalized passing score. If the upper limit of the total score equals the lower limit of the total score, or if the passing score exceeds the range between the lower and upper limits of the total score, no conversion is performed and an anomaly in the scoring rule configuration is generated. Write the effectively generated global weights and normalized passing scores into the structured review dataset.
[0050] At the start of a review batch, the system reads and binds baseline snapshots from batches no later than this batch that correspond to the latest version of the baseline version identifier and have a published status. The baseline version identifier is then written into the structured review dataset. If no matching baseline snapshot exists, a baseline snapshot is generated and bound using a preset initial deviation threshold, a preset initial discrete baseline, and a preset initial confidence multiple.
[0051] The initial deviation threshold is set based on the allowable deviation range of AI expert indicators during the cold start phase; the initial discrete baseline is set based on the allowable dispersion of expert scores during the cold start phase; the target false positive rate is set based on the acceptable proportion of error markings and corresponding review costs in the review management business, and the target false negative rate is set based on the acceptable proportion of abnormal omissions and corresponding business impact in the review management business. The initial confidence multiple is set based on the initial coverage range formed by the mean and standard deviation in historical samples or trial operation samples, combined with the target false positive rate and target false negative rate, serving as the starting point for determining the closed-loop adjustment of the confidence multiple.
[0052] The baseline snapshot includes the initial deviation threshold, the initial discrete baseline, the decision confidence multiple, the AI expert indicator deviation sample statistical parameters for each unified logical indicator node, the discreteness sample statistical parameters distinguished by statistical type, the AI-side deviation confirmation sample statistical information, and the corresponding version information.
[0053] All applications submitted in this batch used the decision parameters and decision confidence multiples from the baseline snapshot.
[0054] S1.2, the review data collection module performs independent inference for the same application project and the same batch of indicator nodes, with a preset number of AI sampling times. Each inference uses the same review analysis language model identifier and the same decoding parameters except for random states.
[0055] The review analysis language model adopts the existing Transformer language model, which includes a lexical embedding layer, a positional encoding layer, a self-attention layer and a feedforward network layer connected in sequence, and an output layer. The input includes the application materials, indicator names, indicator descriptions and scoring rules, and the output is the sub-item scores and sub-item comments corresponding to the indicator nodes of this batch.
[0056] When the reasoning interface of the review analysis language model adopts the random temperature sampling method, the same prompt words are used for each reasoning, different random states are set and temperature sampling is performed according to the preset sampling temperature; when other reasoning methods are adopted, semantically equivalent prompt word templates with different expressions are used for reasoning respectively.
[0057] The number of AI sampling times is set based on the calculation requirements of AI output dispersion and the amount of inference processing. When it is necessary to calculate AI output dispersion, the number of AI sampling times should meet the calculation conditions of sample standard deviation, and an odd number of sampling times should be used so that the median of AI sub-scores corresponds to one actual inference result. When repeated sampling is not enabled, the number of AI sampling times is set to 1.
[0058] The sampling temperature is set according to the output stability and random state differentiation requirements of the review analysis language model. When using temperature sampling, the sampling temperature is greater than 0, and the sampling temperature is kept the same for each independent inference.
[0059] The inference results for each iteration are checked according to the data type and scoring range of the review indicators, and only valid results are retained. When the number of valid results is greater than 0, the valid sub-item scores are sorted, and the median is taken as the AI sub-item score, along with the sub-item comment for that inference iteration as the AI sub-item comment. When the number of valid results is even, the arithmetic mean of the two middle sub-item scores is taken, and the sub-item comment corresponding to the lower score is taken. The AI sub-item scores are converted into AI score components according to the scoring component conversion method in S1.1.
[0060] When the number of valid results is not less than 2, each valid sub-item score is converted into a sampled score component. The sum of squares of the differences between each sampled score component and its arithmetic mean is calculated. The square root of this sum of squares is obtained by dividing it by the number of valid results minus 1. When the number of valid results is 1, the AI output dispersion is empty. When the number of valid results is 0, the AI sub-item score, AI sub-item comment, AI score component, and AI output dispersion are recorded as empty, and the validity flag of the AI review result is set to invalid.
[0061] S1.3, the review data collection module segments the application materials according to title level and paragraph boundaries, forming material fragments with a length not exceeding the preset fragment length limit. Adjacent material fragments retain a preset number of overlapping characters. The fragment length limit is set based on the semantic integrity and retrieval scope of the material fragments, and the number of overlapping characters is set based on the continuity of sentences across material fragments. The scanned text is semantically encoded after character recognition.
[0062] The semantic encoding model employs the existing Transformer encoder, including a lexical embedding layer, a positional encoding layer, a multi-head self-attention layer, a feedforward network layer, and a pooling layer. The input is a sequence of lexical segments of the material fragment, and the output is the semantic vector corresponding to the material fragment. The semantic vector, the application project identifier, the material fragment position, and the fragment text are written into the application material fragment index.
[0063] Experts with valid scoring components are counted in the number of valid experts. Self-consistency checks are performed when the word count of the sub-item comments is not less than the preset minimum word count; AI and expert sub-item scoring checks are performed when there are valid AI review results; outlier identification, inter-expert consistency checks, and scoring scale offset calculations are performed when the number of valid experts is not less than the preset minimum number of valid experts.
[0064] The minimum word count for comments is set based on the text length required to convey a clear evaluation bias, and the minimum number of valid experts is set based on the minimum number of cross-expert statistics to be calculated.
[0065] See appendix Figure 4 In this embodiment, step S2 is executed through the indicator semantic alignment module and includes the following sub-steps: S2.1 The application review system maintains a unified logical indicator node library. The unified logical indicator node library records the unified logical indicator node identifier, indicator name, indicator description, indicator level, and parent unified logical indicator node identifier.
[0066] The indicator semantic alignment module reads the indicator nodes of this batch from the structured review dataset and reads the unified logical indicator nodes of the corresponding indicator level from the unified logical indicator node library. It assembles the indicator name and indicator description into the text to be encoded according to the same field order, deletes control characters, duplicate whitespace characters and invalid format characters according to the preset character cleanup rules, and truncates the text to be encoded according to the preset upper limit of the encoded text length. Then, it inputs the semantic encoding model adopted by S1.3 to obtain the semantic vector.
[0067] Character cleanup rules are set based on the character types and format characteristics in the text to be encoded, and the upper limit of the encoded text length is set according to the input length allowed by the semantic encoding model. A unified encoding method is used to transform indicator nodes from different review batches into the same vector space, enabling indicator nodes with different expressions but similar meanings to be compared.
[0068] The cosine similarity is obtained by dividing the inner product of two semantic vectors by the product of their magnitudes. If a semantic vector is missing or the magnitude of either semantic vector is 0, the cosine similarity is not calculated, and a mapping confirmation is generated.
[0069] Candidate matching is constrained by the indicator hierarchy. First-level indicators only consider first-level unified logical indicator nodes as candidates; second-level indicators only consider second-level unified logical indicator nodes belonging to their parent first-level unified logical indicator node. When a parent first-level indicator is determined to be a temporarily added unified logical indicator node or a newly added unified logical indicator node, its second-level indicators establish corresponding temporarily added unified logical indicator nodes or newly added unified logical indicator nodes under that parent, and do not match with second-level unified logical indicator nodes under other parents.
[0070] A similarity matrix is constructed based on cosine similarity. Node pairs with a cosine similarity less than the preset lower limit of the mapping gray area are not included in the set of matchable node pairs. Under the mutual exclusion constraint within the same batch, bipartite graph maximum weight matching is performed on the set of matchable node pairs. The sum of the cosine similarities of all matched node pairs is used as the total weight to obtain the matching scheme with the maximum total weight. Within the same batch, an indicator node can be mapped to at most one unified logical indicator node, and a unified logical indicator node can accept at most one indicator node in this batch.
[0071] The mapping result is determined when the cosine similarity of the matching node pair is not less than the preset mapping similarity limit; when the corresponding unified logical index node is not the highest similarity candidate, but the cosine similarity is still not less than the mapping similarity limit, the determination method is recorded as the second best candidate mapping.
[0072] When the cosine similarity is not less than the lower limit of the mapping gray area and less than the mapping similarity limit, a mapping confirmation item is generated; when there are no candidates that reach the lower limit of the mapping gray area, it is determined to be a new unified logical indicator node, the corresponding unified logical indicator node identifier is generated, and the unified logical indicator node identifier, indicator name, indicator description, indicator level and parent unified logical indicator node identifier are written into the unified logical indicator node library; when no mapping result is obtained due to mutual exclusion constraints in the same batch, but there are candidates that reach the lower limit of the mapping gray area and have been occupied, a mapping confirmation item is also generated.
[0073] Before the mapping confirmation result is returned, a temporary unified logical indicator node identifier is generated for the indicator nodes of this batch, and the corresponding indicator name, indicator description, indicator level and parent unified logical indicator node identifier are recorded. After the review and management personnel confirm, the indicator nodes of this batch are mapped to the confirmed unified logical indicator nodes, or the temporarily added unified logical indicator nodes are confirmed as newly added unified logical indicator nodes and written into the unified logical indicator node library, while the historical sample affiliation is updated.
[0074] The mapping similarity limit is set based on the mapping requirements of the same semantic indicators and the error merging limit, and the lower limit of the mapping gray area is used to limit the scope of manual confirmation. The initial deviation threshold and the initial discrete baseline are used as the judgment parameters for this batch for temporarily added unified logical indicator nodes and newly added unified logical indicator nodes.
[0075] In step S2.2, the indicator semantic alignment module writes the mapping relationship, cosine similarity, determination method, effective time, expiration time, and confirmation status into the indicator mapping relationship table. Determination methods include automatic mapping, suboptimal candidate mapping, manual confirmation mapping, adding new nodes, and post-merging. When subsequent batches encounter the same indicator node identifier, a valid mapping relationship is read; if no valid mapping relationship exists, step S2.1 is re-executed.
[0076] After mapping, invalid expert scoring components are excluded based on data validity identifiers, and invalid AI scoring components are excluded based on AI review result validity identifiers. Valid expert scoring components, valid AI scoring components, expert comments, AI comments, AI output dispersion, review subject identifier, review subject role, and expert weights are aggregated to the corresponding unified logical indicator nodes. The global weights of the indicator nodes for this batch are associated with the mapped unified logical indicator nodes, and normalization is associated with this review batch through the score line. Subsequent statistics, verification, attribution, and maintenance of valid historical samples are all performed on the unified logical indicator nodes.
[0077] Once the number of valid historical score component samples for newly added unified logical indicator nodes reaches the preset minimum sample size, the cosine similarity is recalculated within the same indicator level; secondary unified logical indicator nodes should also have the same parent primary unified logical indicator node.
[0078] The historical score component mean values of the original unified logical indicator node and the candidate merge node are calculated based on their respective valid historical score component samples. A node merging event is generated when the cosine similarity is not less than the mapping similarity limit, and the absolute value of the difference between the historical score component mean values of the two nodes is not greater than the preset scale offset limit. The minimum sample size is set based on the statistical stability requirements of the historical score component mean values; the scale offset limit is set based on the negligible difference in score scale.
[0079] After the review and management personnel confirm the merger, the valid historical samples of the original node will be assigned to the target node, the original mapping relationship will be written to the expiration time, and a mapping relationship with the determination method of post-merging will be established; if the merger is not confirmed, the two unified logical indicator nodes will continue to be stored independently.
[0080] S2.3, the indicator semantic alignment module, for sub-comments with a word count not less than the minimum comment word count, inputs the sub-comment, indicator name, and indicator description into the review analysis language model used in S1.2, and outputs three types of confidence scores: positive, neutral, and negative, according to fixed fields. When all three types of values are within the closed interval of 0 to 1 and the sum is greater than 0, each value is divided by the sum of the three types of values to make the sum of the three types of confidence scores equal to 1. If there are missing fields, incorrect data types, values exceeding the closed interval of 0 to 1, or the sum equals 0, the review analysis language model is called again according to the preset number of parsing retries.
[0081] Positive, neutral, and negative values are quantified as +1, 0, and -1 respectively. These are then multiplied by their corresponding confidence levels and summed to obtain the semantic tendency component within the closed interval of -1 to +1. The maximum value among the three confidence levels is used as the discrimination confidence level. The number of parsing retries is set based on the error correction requirements of the structured output, and the lower limit of the tendency confidence level is set based on the minimum distinguishable degree of the evaluation tendency. If no valid result can be obtained after parsing retries, or if the discrimination confidence level is less than the lower limit of the tendency confidence level, the semantic tendency component is recorded as empty. The semantic tendency component is also recorded as empty if the number of words in the sub-comment is less than the minimum number of words in the comment.
[0082] The rating component and semantic tendency component are combined into a review vector according to a fixed field order. The review vector is then associated with the review subject identifier and written into the corresponding unified logical indicator node. When the semantic tendency component is empty, a null value identifier is retained, and subsequent validations dependent on the semantic tendency component are not performed; the rating component still participates in the numerical validation.
[0083] See appendix Figure 5In this embodiment, step S3 is performed by the scoring scale correction module and includes the following sub-steps: S3.1, the scoring scale correction module reads the scoring components of each expert from the unified logical indicator node and filters the unified logical indicator nodes whose number of valid experts is not less than the minimum number of valid experts. For each target expert, the node deviation is calculated using the node deviation formula: ; In the formula, For experts At the unified logical indicator node On the node deviation; For experts At the unified logical indicator node The rating components on; For experts whose node deviations need to be determined; To exclude experts Other experts besides; This indicates taking the median; Represents a set; This indicates that the elements of a set must satisfy the conditions that follow.
[0084] When calculating the median, the data are arranged in ascending order. If the number of data points is odd, the middle value is used; if the number of data points is even, the arithmetic mean of the two middle values is used. Using the median of the remaining expert rating components as a reference can reduce the impact of a single outlier rating on node bias. Node bias represents the target expert's rating position on a single unified logical indicator node; when multiple unified logical indicator nodes show consistent node bias in the same direction, this consistent bias is used to determine the rating scale offset.
[0085] The minimum deviation sequence length is set based on the stability calculation requirements of the batch offset. When the batch offset sequence length is not less than the minimum deviation sequence length, the median of the batch offset sequence is taken as the batch offset, and the number of node offsets that are both positive or both negative with the batch offset is divided by the batch offset sequence length to obtain the batch's directional consistency ratio. When the batch offset is equal to 0, node offsets are not included in the positive or negative consistency count. When the batch offset sequence length is less than the minimum deviation sequence length, the batch offset and the batch's directional consistency ratio are not estimated.
[0086] The application review system maintains a scoring scale profile based on expert identifiers; when no historical review batch data exists, the corresponding expert's historical deviation sequence is empty. The scoring scale profile saves the historical deviation sequence of each expert according to the preset number of profile window batches, and records the review batch identifier, unified logical indicator node identifier, and validity status. Valid historical deviation sequences within the profile window are merged, and the median of all valid node deviations is taken as the historical offset; the number of node deviations that are both positive or both negative with the historical offset is divided by the total number of valid node deviations to obtain the historical direction consistency ratio. The number of profile window batches is set based on the stability of historical data and the range of reflection of recent scoring changes.
[0087] When both the batch offset and the historical offset are valid, the scale offset formula is used to determine the scoring scale offset: ; In the formula, For experts The rating scale offset; For experts The offset within the batch; For experts Historical offset; This is the preset weight for this batch.
[0088] Multiply the current batch's directional consistency ratio by the current batch's weight, multiply the historical directional consistency ratio by 1 and subtract the current batch's weight, then add the two products together to obtain the deviation directional consistency ratio.
[0089] The weights for this batch are set based on the sufficiency of the sample, statistical stability, and recent changes in the scoring scale of the current batch deviation sequence and the effective historical deviation sequence. The values are in a closed interval of 0 to 1. Larger values are used when the statistical stability of the current batch deviation sequence is high, and smaller values are used when the statistical stability of the effective historical deviation sequence is high.
[0090] When the historical offset is empty, the scoring scale offset is taken as the batch offset, and the deviation direction consistency ratio is taken as the batch direction consistency ratio; when the deviation sequence length of the current batch is less than the minimum deviation sequence length, the scoring scale offset is taken as the historical offset, and the deviation direction consistency ratio is taken as the historical direction consistency ratio; when both the batch offset and the historical offset are empty, the corrected scoring component is equal to the scoring component.
[0091] S3.2, the scoring scale correction module compares the scoring scale offset with the scale offset limit, and compares the deviation direction consistency ratio with the direction consistency ratio limit. If the absolute value of the scoring scale offset is greater than the scale offset limit, and the deviation direction consistency ratio is greater than the preset direction consistency ratio limit, scoring scale correction is performed on the expert. The direction consistency ratio limit is set based on the minimum percentage of same-direction node deviations required to determine if the scoring scale offset has systematic characteristics.
[0092] When the absolute value of the scoring scale offset is not greater than the preset overcorrection upper limit, the scoring scale offset is subtracted from the scoring components of each unified logical indicator node for that expert. When the absolute value of the scoring scale offset is greater than the overcorrection upper limit, if the scoring scale offset is positive, the overcorrection upper limit is subtracted from the scoring components; if the scoring scale offset is negative, the overcorrection upper limit is added to the scoring components, and an anomaly warning is generated. The overcorrection upper limit is set based on the maximum allowable scoring scale offset for the review process, as well as the distribution of offset amplitudes corresponding to differences in scoring scales, differences in indicator understanding, and scoring entry errors in historical reviews. A result less than 0 is taken as 0, and a result greater than 1 is taken as 1, yielding the corrected scoring components.
[0093] When the scoring scale offset or the proportion of consistency in the deviation direction does not meet the correction conditions, the corrected scoring component is equal to the original scoring component. The scoring scale offset, the proportion of consistency in the deviation direction, whether the correction upper limit has been reached, and the corrected scoring component are associated with the expert identifier and the unified logical indicator node identifier, and written into the corresponding unified logical indicator node.
[0094] The scoring scale correction uses the expert scoring components as the processing object, and retains the original values of the AI scoring components during the correction process. The deviation sequence of this batch, the review batch identifier, and the corresponding unified logical indicator node identifier are written into the scoring scale profile, and the validity status of the deviation sequence of this batch is set to pending confirmation. The validity status of the deviation sequence of this batch and the method of writing it into the valid historical deviation sequence are determined according to the review processing rules in S6.2.
[0095] See appendix Figure 6 In this embodiment, step S4 is executed by the consistency verification module and includes the following sub-steps: S4.1 The consistency verification module reads the baseline snapshot bound to this batch and the review vectors of each review subject on the unified logical indicator node. When the score component is greater than the preset high score interval limit and the semantic tendency component is less than the negative of the preset tendency limit value, or when the score component is less than the preset low score interval limit and the semantic tendency component is greater than the tendency limit value, it is marked as a self-consistency conflict indicator; when the semantic tendency component is empty, this verification is not performed.
[0096] The boundaries of the high-score and low-score intervals are set according to the division between high-score and low-score evaluations in the review process, and the tendency threshold value is set according to the absolute value of the semantic tendency component required for the comments to have a clear affirmative or negative tendency.
[0097] Input the AI sub-comments, indicator names, and indicator descriptions into the review analysis language model used in S1.2. The model outputs factual statements and statement types, performing material support verification only on documented and numerical data. If parsing fails, it will be re-parsed according to the number of retries. If it still fails, the material support verification for the corresponding factual statement will not be performed.
[0098] The factual statement is input into the semantic encoding model adopted in S1.3 to obtain the semantic vector of the factual statement. The cosine similarity between this semantic vector and the semantic vectors of each material fragment in the application material fragment index is calculated. Candidate supporting fragments and the highest similarity of candidate supporting fragments are determined according to the cosine similarity from high to low. When the highest similarity of a candidate supporting fragment is less than the preset material support similarity threshold, the factual statement is marked as having no material support comment. The material support similarity threshold is set based on the minimum similarity level that allows a semantic correspondence between the factual statement and the material fragment.
[0099] When the highest similarity of a candidate supporting segment reaches the material support similarity threshold, the numerical format, unit of measurement, statistical period format, and corresponding object name are standardized according to preset field standardization rules. These rules are set based on the unit of measurement, numerical precision, period format, and object name used in the review indicator configuration. If the absolute value of the numerical difference is not greater than the preset numerical tolerance, and all other items are consistent, it is determined that there is material support; otherwise, it is marked as a material support conflict comment. The numerical tolerance is set based on the measurement precision and allowable error of the corresponding field.
[0100] S4.2 When the number of valid experts for a unified logical indicator node is not less than the minimum number of valid experts, the consistency verification module calculates the absolute value of the difference between the corrected score component of each expert and the median of the corrected score components of the other experts to obtain the consistency distance.
[0101] The sum of the median of all consensus distances and the absolute deviation of twice the median is used as the threshold for outlier experts. Experts exceeding this threshold are considered outlier candidates. The upper limit for the number of outliers is the smaller of the number of valid experts divided by 3 and rounded down, and the number of valid experts minus the minimum number of valid experts, and it is not less than 0. When there are too many outlier candidates, they are selected in descending order of consensus distance; the calculation is repeated after removal until there are no new outlier candidates or the upper limit for the number of outliers is reached, and the outlier expert is identified.
[0102] After removing outlier experts, the arithmetic mean of the remaining expert-corrected score components is used as the robust expert mean. When a unified logical indicator node has valid AI review results, the AI expert indicator deviation is obtained by subtracting the robust expert mean from the AI score component. When the baseline snapshot has the sample statistical parameters of the AI expert indicator deviation for the corresponding unified logical indicator node (i.e., the corresponding historical parameters), the dynamic deviation threshold used for this verification is calculated using the dynamic threshold formula: ; In the formula, To unify logical indicator nodes The dynamic deviation threshold; To unify logical indicator nodes Historical AI expert metric bias; This represents the mean of the absolute values of the historical AI expert indicator deviations. This represents the corresponding sample standard deviation; To determine the confidence level; This is the preset lower threshold. This indicates taking the maximum value; This indicates taking the absolute value.
[0103] The lower limit of the threshold is set based on the lowest magnitude of the difference that can be considered substantial in the normalized scoring space and the concentration of the absolute values of historical AI expert indicator deviations. The confidence factor is read from the baseline snapshot bound to this batch, and subsequent values are determined according to the closed-loop adjustment rules in S6.3. If the baseline snapshot does not have corresponding statistical parameters for AI expert indicator deviation samples, the initial deviation threshold in the baseline snapshot bound to this batch is used.
[0104] S4.3, the consistency verification module uses the actual threshold formula when the AI output dispersion is not empty: ; In the formula, The threshold for determining the actual deviation; The dynamic deviation threshold used in this verification; This is a preset AI instability relaxation factor; Output the discreteness for AI; To determine the threshold relaxation value.
[0105] The AI instability relaxation coefficient is set based on the relaxation range of AI output dispersion, so that it is on the same order of magnitude as the decision confidence factor, and the decision threshold relaxation value covers the main output fluctuation range of the AI review results.
[0106] When the AI output dispersion is empty, the actual deviation judgment threshold is the dynamic deviation threshold used in this verification. When the absolute value of the AI expert indicator deviation is greater than the actual deviation judgment threshold, it is marked as a difference indicator. When the AI output dispersion is greater than the preset AI stability limit, it is marked as an unstable AI output indicator. A sample exclusion flag is set and the corresponding data is excluded from the AI expert indicator deviation sample. At the same time, an AI re-review task is generated.
[0107] The AI re-evaluation task is re-executed according to S1.2, updating the AI sub-scores, AI sub-comments, AI score components, and AI output dispersion, and recalculating the actual deviation judgment threshold and AI expert index deviation. The judgment results of the difference index are updated based on the recalculation results.
[0108] If the updated AI output dispersion is not empty and does not exceed the AI stability limit, cancel the AI output instability indicator and its sample exclusion flag, and end the AI re-review task; if the updated AI output dispersion is empty or still exceeds the AI stability limit, and the cumulative number of AI re-review tasks has not reached the preset number of AI re-reviews, continue to execute the AI re-review task; if a stable AI output dispersion is still not obtained after reaching the number of AI re-reviews, retain the AI output instability indicator and its sample exclusion flag, and provide the corresponding AI output instability indicator as a marked item to S6.1 for review priority ranking.
[0109] The AI stability limit is set based on the allowable fluctuation range of AI repeated sampling scores; the number of AI re-reviews is set based on the number of times the stability of the AI output needs to be repeatedly confirmed and the amount of inference processing required. When the semantic tendency component of the AI review result has the opposite sign to the semantic tendency component of the expert review result that is more than half the number of valid experts, and the absolute value of the semantic tendency component of the AI review result and the absolute value of the semantic tendency component of each corresponding expert review result are both greater than the tendency limit value, it is marked as a tendency conflict index.
[0110] S4.4, for unified logical indicator nodes where the number of valid experts is not less than the minimum number of valid experts, the consistency verification module calculates the arithmetic mean and population standard deviation based on the corrected score components of each expert at that node. When the arithmetic mean is not less than a preset lower limit of the mean, the coefficient of variation is obtained by dividing the population standard deviation by the arithmetic mean, and the coefficient of variation is used as the dispersion statistic; when it is lower than the lower limit of the mean, the population standard deviation is used as the dispersion statistic. The lower limit of the mean is set to avoid amplifying the coefficient of variation when the arithmetic mean is close to 0. The coefficient of variation sample and the population standard deviation sample are saved and compared separately, and the dispersion sample statistical parameters corresponding to the current dispersion statistic type in the baseline snapshot bound to this batch are read to calculate the dynamic dispersion baseline: ; In the formula, For dynamic discrete baselines; This is a statistic on the degree of dispersion. The mean of the historical dispersion statistic; This represents the corresponding sample standard deviation; The same confidence factor as the dynamic threshold formula; This is the preset lower limit of the discrete baseline.
[0111] The lower limit of the discrete baseline is set based on the concentration of historical dispersion statistics to avoid the dynamic discrete baseline being too small when the historical dispersion statistics are highly concentrated.
[0112] When the corresponding dispersion sample statistical parameters are missing, the initial dispersion baseline is used. When the current dispersion statistic exceeds the dynamic dispersion baseline, it is marked as a divergence index.
[0113] For a unified logical index node marked as a divergence index, if the number of valid experts is not less than 4, the corrected score components are arranged from smallest to largest. The split positions that make the number of experts on both sides not less than 2 are enumerated. The sum of squares of the difference between the corrected score component on each side and the arithmetic mean on that side are calculated respectively. The split position with the smallest sum of squares on both sides is used to form two clusters. If there are multiple split positions with the same sum of squares, the split position with the smallest position number is selected.
[0114] The arithmetic mean of the corrected scores on both sides is used as the cluster centers of the two clusters. When the distance between the cluster centers is greater than the preset cluster distance limit, the divergence pattern is recorded as a polarization pattern. When the number of valid experts is less than 4, the polarization pattern determination is skipped. When it is not determined to be a polarization pattern but there are outlier indicators, the divergence pattern is recorded as an individual expert outlier pattern. When it is not determined to be a polarization pattern and there are no outlier indicators, only the divergence index is retained.
[0115] The cluster spacing limit is set based on the minimum range that allows for substantial differences in scores between expert scoring groups.
[0116] For each expert, the sum of the global weights of the effective unified logical indicator nodes is calculated; if this sum is lower than the preset minimum weight coverage, no aggregate conclusion is generated. The minimum weight coverage is set based on the representativeness of the effective unified logical indicator nodes to the overall review indicators; when the sum reaches the minimum weight coverage, the global weights of the effective unified logical indicator nodes are renormalized, and the scoring components and the corrected scoring components are weighted according to the renormalized global weights of the effective nodes to obtain the original aggregate total score and the corrected aggregate total score.
[0117] The two aggregate totals are compared with the normalized scores through the fraction line to obtain the original aggregate conclusion and the corrected aggregate conclusion.
[0118] The final review conclusion is determined based on the submitter's role: it is either an individual final review or a team leader final review. Individual final reviews compare the corresponding expert's original and corrected aggregated conclusions with the final review conclusion. Team leader final reviews calculate the sum of the expert weights for experts for whom original and corrected aggregated total scores have been generated. If the sum of expert weights is greater than 0, each expert's weight is divided by the sum of expert weights to obtain a normalized expert weight. This normalized weight is then applied to each expert's original and corrected aggregated total scores and compared with the normalized weights using a passing score line to obtain the review group's original and corrected aggregated conclusions. If the sum of expert weights is 0, an anomaly is generated in the scoring rule configuration, and no review group aggregated conclusion is generated.
[0119] When both the original aggregation conclusion and the corrected aggregation conclusion contradict the final review conclusion, it is judged as a logical conflict of conclusions; when the two are inconsistent, it is recorded as a scale-sensitive aggregation conclusion; when the final review conclusion is to postpone the review, a postponement notice is generated.
[0120] When logical conflicts exist in the conclusions, the final review opinion text, review indicator names, and review indicator descriptions are input into the review analysis language model used in S1.2. The output is the review indicator pointed to by the final review opinion text, the affirmative or negative polarity, and the confidence level. The output review indicators are then mapped to unified logical indicator nodes according to the indicator semantic alignment method in S2.1. Analysis results with a confidence level not less than the lower limit of the propensity confidence level and a mapping similarity not less than the mapping similarity limit are used to determine whether the difference in conclusions is reasonable. Individual final reviews use the arithmetic mean of the corrected score components of each node for the corresponding expert as a reference, while group leader final reviews use the arithmetic mean of the robust expert mean for each node as a reference, forming a low-scoring indicator set and a high-scoring indicator set according to the preset quantile range.
[0121] The quantile range is set based on the minimum range at which a node's score significantly differs from a reference value, resulting in a high or low score. A reasonable difference is determined when both the original and corrected aggregation conclusions are "pass," the final conclusion is "fail," and the negative statement points to the low-scoring indicator set; or when both the original and corrected aggregation conclusions are "fail," the final conclusion is "pass," and the positive statement points to the high-scoring indicator set. Other situations, including those where parsing fails, are identified as logical conflicts requiring further review.
[0122] Self-consistency conflict indicators, comments without supporting materials, comments with conflicting supporting materials, AI output instability indicators, difference indicators, tendency conflict indicators, divergence indicators, and logical conflicts pending review are respectively designated as marked items; review data that have completed corresponding verification but have not formed any of the aforementioned marked items are designated as unmarked items.
[0123] See appendix Figure 7 In this embodiment, step S5 is executed through the conflict attribution analysis module and includes the following sub-steps: S5.1 The conflict attribution analysis module reads the rating scale offset, the proportion of consistent deviation direction, the consistency distance, the self-consistency conflict index, the unsupported comments, the supported conflict comments, the difference index, the tendency conflict index, the AI output instability index, the divergence index, the logic conflict to be reviewed, and the outlier expert identifier, and executes the attribution rules respectively.
[0124] When the absolute value of the rating scale offset is greater than the scale offset limit, and the proportion of deviations in the same direction is greater than the proportion of directions in the same direction limit, a rating scale difference is identified. This attribution corresponds to a situation where experts consistently exhibit rating offsets in the same direction across multiple unified logical indicator nodes.
[0125] When marked as an unstable AI output indicator, it indicates that the AI output is unstable; when marked as a comment without supporting material or a comment with conflicting supporting material, it indicates that the AI statement lacks supporting material.
[0126] For discrepancy indicators or conflict-prone indicators, the AI-side bias confirmation sample statistics of the corresponding unified logical indicator node are read from the baseline snapshot bound in this batch. The AI-side bias confirmation sample statistics are formed by historical records that have been manually reviewed and whose sources of discrepancy are clear, including the number of times it has been confirmed as AI-side bias and the total number of times manual review has been completed.
[0127] The number of AI-confirmed biases is added to a preset smoothing coefficient to obtain the confirmation number correction value; the total number of manual reviews is added to twice the smoothing coefficient to obtain the review number correction value; the confirmation number correction value is divided by the review number correction value to obtain the AI bias confirmation rate. The smoothing coefficient is set based on the expected effective sample size of the AI bias confirmation samples and the weight of prior information in the calculation of the AI bias confirmation rate, taking a number greater than 0. A larger value is used when the expected effective sample size is small or the weight of prior information is high, and a smaller value is used when the expected effective sample size is large.
[0128] If the total number of manual reviews completed is not less than the preset minimum number of confirmed samples, and the AI side bias confirmation rate is greater than the preset AI bias confirmation rate threshold, then the AI applicability difference is hit. The minimum number of confirmed samples is set based on the minimum statistical quantity of AI side bias confirmation rate, and the AI bias confirmation rate threshold is set based on the judgment requirement that the proportion of AI side bias in historical review results reaches the majority.
[0129] If the total number of manual reviews completed is less than the minimum number of confirmed samples, the AI-side deviation confirmation samples of each secondary unified logical indicator node under the same primary unified logical indicator node are merged to form a parent merged confirmation sample, and the AI-side deviation confirmation rate is recalculated according to the same process. If the parent merged confirmation sample is still less than the minimum number of confirmed samples, the AI applicability difference is not hit.
[0130] S5.2, the conflict attribution analysis module continues to execute attribution rules that must be judged in conjunction with the current consistency check results. When a conflict is marked as a self-consistent conflict indicator, it indicates inconsistency in the internal statements of the target entity. This attribution corresponds to the situation where the scoring component and semantic tendency component of the same review entity on the same unified logical indicator node are in opposite directions.
[0131] An expert's failure to identify a scoring scale difference is considered a success if they have an outlier expert identifier only on one unified logical indicator node, and the consensus distance on each of the remaining valid unified logical indicator nodes is no greater than the median consensus distance of all experts on that node. If no other valid unified logical indicator nodes are available for comparison, the expert's failure to identify a scoring scale difference is considered a failure. This rule distinguishes between scoring deviations on a single unified logical indicator node and scoring scale shifts that persist across unified logical indicator nodes.
[0132] When a conclusion is marked as having a logical conflict pending review, it indicates a logical conflict in the conclusion; if the aforementioned attribution rules' difference indicators, tendency conflict indicators, or divergence indicators are not met, it indicates an unclassified difference.
[0133] After completing all attribution rules, the hit results are ranked in the following order: differences in scoring scales, unstable AI output, lack of supporting materials for AI statements, differences in AI applicability, inconsistencies in internal statements, differences in indicator understanding, logical conflicts in conclusions, and differences in unclassified results. The hit result at the top of the list is determined as the primary attribution label, and the remaining hit results are determined as secondary attribution labels. The primary and secondary attribution labels are associated with the application project identifier, review batch identifier, review entity identifier, and unified logical indicator node identifier, and provided to the verification result generation module.
[0134] See appendix Figure 8 In this embodiment, step S6 is executed by the verification result generation module and includes the following sub-steps: S6.1, the verification result generation module summarizes the self-consistency conflict indicators, comments without supporting materials, comments with supporting materials, AI output instability indicators, difference indicators, tendency conflict indicators, divergence indicators, divergence patterns, outlier expert identifiers, scoring scale offsets, logic conflicts to be reviewed, mapping confirmation items, node merging items, main attribution labels and additional attribution labels, as well as scoring rule configuration anomalies, scoring scale anomaly prompts, scale-sensitive aggregation conclusions and deferred discussion prompts, and associates the baseline version identifier, review analysis language model identifier, semantic encoding model identifier and prompt word version identifier to generate a structured consistency verification report.
[0135] The review priority score is obtained by multiplying the over-limit ratio, the global weight of the corresponding unified logical indicator node, and the weight of the attribution category corresponding to the main attribution label. The over-limit ratio of the difference indicator is the absolute value of the AI expert indicator deviation divided by the actual deviation judgment threshold; the divergence indicator is the dispersion statistic divided by the dynamic dispersion baseline; the self-consistency conflict indicator is the absolute value of the semantic tendency component divided by the tendency limit value; and the AI output instability indicator is the AI output dispersion divided by the AI stability limit. The "no supporting material" comment is calculated by dividing the material support similarity limit by the highest similarity of the candidate supporting segments. If the highest similarity is not greater than a preset non-zero calculation lower limit, the non-zero calculation lower limit is used as the denominator. The non-zero calculation lower limit is set to a number greater than 0 based on the numerical precision of semantic similarity to avoid a denominator of 0 in the calculation of the review priority score. The material support conflict assessment is calculated by dividing the highest similarity of the candidate supporting segments by the material support similarity limit. For marked items with pending logical conflicts and those for which the excess ratio cannot be calculated, the excess ratio is set to 1; the global weight for pending logical conflicts is set to 1.
[0136] The weight of the attribution category is set according to the degree of influence of the attribution category on the review conclusion. For example, logical conflict of conclusion is 1.0, lack of material support for AI expression is 0.95, inconsistency in internal expression of the subject is 0.9, difference in scoring scale is 0.8, difference in AI applicability is 0.7, difference in indicator understanding is 0.6, difference in unclassified is 0.5, and instability of AI output is 0.3.
[0137] The marked items are ranked from highest to lowest according to their review priority score. Review work orders are generated sequentially for each marked item, with the number of work orders limited to a preset upper limit. The upper limit is set based on the review workload that the review manager can complete in a single review batch. Mapped confirmation items and node merging items are not included in the review priority ranking and are provided to the review manager for confirmation as items to be confirmed in the structured consistency verification report.
[0138] S6.2, the verification result generation module divides the secondary unified logical indicator nodes under the same primary unified logical indicator node into the same statistical category, and randomly selects the first group of sampling verification work orders and the second group of sampling verification work orders from the marked items and the unmarked items, respectively, according to the statistical category and the preset sampling verification ratio. Verification work orders generated according to verification priority, the first group of sampling verification work orders, and the second group of sampling verification work orders are generated separately. The sampling verification ratio is set based on the estimated requirements of the measured false positive rate and the measured false negative rate, as well as the verification workload.
[0139] Receive the review and confirmation results submitted by the review and management personnel, including whether there is an actual anomaly, the source of the difference, and whether the difference is a reasonable conclusion.
[0140] The valid historical rating component sample only includes expert rating components that are marked as valid and do not contain any verified errors in rating entry or differences in indicator interpretation.
[0141] The AI expert indicator deviation sample only includes data that is not marked as an unstable AI output indicator and has a number of valid experts not less than the minimum number of valid experts; under the aforementioned conditions, the data should also be reviewed and confirmed to have no actual anomalies or reasonable differences in conclusions, or belong to unmarked items that have been reviewed and confirmed to have no actual anomalies.
[0142] The dispersion sample only includes data with a valid number of experts not less than the minimum valid number of experts, and without confirmed scoring errors or differences in indicator interpretation; disagreements that are confirmed to be reasonable differences in conclusions can be included. The AI-side bias confirmation sample only includes discrepancy indicators or tendency conflict indicators that have completed manual review and whose sources of difference are clearly identified; the review and confirmation results of the first group of sampling review work orders are used to calculate the measured false positive rate, and the review and confirmation results of the second group of sampling review work orders are used to calculate the measured false negative rate.
[0143] Various sample types are maintained in a sliding window manner according to the preset batch number of historical sample windows. The batch number of historical sample windows is set based on the sufficiency of valid historical samples and the requirement for the judgment parameters to follow recent review changes. The dispersion sample retains the statistical type of the dispersion coefficient sample or the population standard deviation sample. Node biases that are confirmed to be scoring entry errors or differences in indicator interpretation after review are set to invalid and excluded from the valid historical bias sequence; node biases that are confirmed to have no scoring entry errors or differences in indicator interpretation are set to valid and added to the corresponding expert's valid historical bias sequence.
[0144] In S6.3, the verification result generation module uses the expert rating component as the valid historical sample value for the valid historical rating component sample, and saves the sample quantity and the sum of sample values respectively. The mean of the historical rating component is obtained by dividing the sum of sample values by the sample quantity, which is used for the node merging judgment in S2.2.
[0145] For AI expert indicator deviation samples, the absolute value of the AI expert indicator deviation is used as the valid historical sample value; for dispersion samples, the dispersion statistic is used as the valid historical sample value, and a distinction is made between the dispersion coefficient sample and the population standard deviation sample. The sample size, sum of sample values, and sum of squared sample values are stored for both the AI expert indicator deviation sample and the various dispersion samples, respectively, to calculate the corresponding sample mean and sample standard deviation.
[0146] For AI-biased samples, save the number of times they were confirmed as AI-biased and the total number of times manual review has been completed. The corresponding number is added when a sample is included and deducted when a sample is removed from the historical sample sliding window.
[0147] For AI expert indicator bias samples and dispersion samples, when the number of valid historical samples is 0, both the valid historical sample mean and sample standard deviation are empty; when the number of valid historical samples is not less than 1, the valid historical sample mean is obtained by dividing the sum of valid historical sample values by the number of valid historical samples; when the number of valid historical samples is not less than 2, the sample variance is obtained by subtracting the square of the sum of the squares of the valid historical sample values from the sum of the squares of the valid historical sample values, dividing by the number of valid historical samples, and then dividing by the number of valid historical samples minus 1. The square root of this square root is then used to obtain the sample standard deviation. Floating-point calculations are used to take a value of 0 when the sample variance is less than 0.
[0148] When the number of AI expert indicator deviation samples is not less than the minimum sample size, the AI expert indicator deviation sample statistical parameters of the corresponding unified logical indicator node are updated according to the samples of this type; when the number of dispersion samples is not less than the minimum sample size, the dispersion sample statistical parameters are updated according to the samples of the corresponding statistical type.
[0149] When the number of corresponding samples is less than the minimum sample size, similar samples from all secondary unified logical indicator nodes under the same primary unified logical indicator node are merged. If the number of similar samples after merging is still less than the minimum sample size, the corresponding initial deviation threshold or initial discrete baseline is retained in the next baseline snapshot. The updated sample statistical parameters are written to the next baseline snapshot in the pending release state for subsequent review batches to calculate the dynamic deviation threshold and dynamic discrete baseline according to the formulas in S4.2 and S4.4.
[0150] Statistical analysis was conducted according to the statistical categories defined in S6.2. From the first group of sampled review work orders, the second group of sampled review work orders, marked items, and unmarked items, items whose judgments were made using dynamic deviation thresholds or dynamic discrete baselines were selected as the statistical objects for adjusting the judgment confidence multiples.
[0151] The number of samples in the first group that, after review, are confirmed not to constitute actual anomalies is divided by the total number of samples in the first group for that category, yielding the measured false positive rate for that category. The proportion of samples in the second group that constitute actual anomalies is multiplied by the total number of unlabeled samples in that category, yielding the estimated number of missed detections for that category. The estimated number of missed detections is then divided by the sum of the estimated number of missed detections and the number of labeled samples in that category that, after review, constitute actual anomalies within the same historical sample sliding window, yielding the measured false negative rate for that category.
[0152] When the first group of samples has a sample size of 0, the measured false positive rate is empty; when the second group of samples has a sample size of 0, the estimated number of missed detections is not calculated, and the measured missed detection rate is empty; when the sum of the estimated number of missed detections and the number of marked statistical objects that have been verified to constitute actual anomalies is 0, the measured missed detection rate is empty.
[0153] If the sample size in the first or second group is less than the preset minimum statistical sample size, or if any statistical result is empty, the confidence factor remains unchanged. The minimum statistical sample size is set based on the statistical stability requirements of the measured false positive rate and the measured false negative rate.
[0154] When the measured false positive rate is greater than the preset target false positive rate and the measured false negative rate is not greater than the preset target false negative rate, the confidence factor is increased by a preset adjustment step size; when the measured false negative rate is greater than the target false negative rate and the measured false positive rate is not greater than the target false positive rate, the confidence factor is decreased by an adjustment step size; when both are not greater than or both are greater than the corresponding target value, the confidence factor remains unchanged; when both are greater than, a verification dimension insufficiency prompt is generated and written to the structured consistency verification report. The step size is set based on the sensitivity between changes in the confidence fold and changes in the measured false positive rate and measured false negative rate in historical samples or trial operation samples; the lower and upper confidence fold limits are set based on the effective range of values for the confidence fold in historical samples or trial operation samples.
[0155] If no historical confidence multiple adjustment results exist, the initial confidence multiple determined in S1.1 shall be used. If the adjusted confidence multiple is less than the lower confidence multiple limit, the lower confidence multiple limit shall be used; if it is greater than the upper confidence multiple limit, the upper confidence multiple limit shall be used.
[0156] Under the same first-level unified logical indicator node, all second-level unified logical indicator nodes share the same judgment confidence multiple, and the adjustment result is written to the next baseline snapshot.
[0157] S6.4 The verification result generation module uses the baseline snapshot bound to this batch during the execution of this batch. The next baseline snapshot remains in a pending-release state before release and is only available for review batches after this batch. The preset end condition is that all submitted projects in this batch have completed consistency verification, and the system time of the application review system is greater than or equal to the review deadline; wherein, the review deadline is configured by the review administrator in the review batch information of the application review system, indicating the deadline for receiving the review confirmation results for this batch.
[0158] Once the termination conditions are met, the final set of valid historical samples will be determined based on the review confirmation results received up to the review deadline. Data corresponding to review work orders for which no review confirmation results have been submitted will not be included in the final set of valid historical samples.
[0159] Update the statistical information of the effective historical scoring component samples, the statistical parameters of the AI expert indicator deviation samples, the statistical parameters of the dispersion samples, and the statistical information of the AI side bias confirmation samples based on the final effective historical sample set. Then, recalculate the measured false positive rate and the measured false negative rate according to the statistical method in S6.3 to determine the final judgment confidence multiple.
[0160] The next baseline snapshot is generated based on the updated statistical information and the final decision confidence multiple. The initial deviation threshold, initial discrete baseline, decision confidence multiple, statistical information of effective historical scoring component samples, statistical parameters of AI expert indicator deviation samples, statistical parameters of dispersion samples, and statistical information of AI-side deviation confirmation samples are written in. The application review system generates a new baseline version identifier, sets the effective batch as the next review batch after this batch, and writes the review analysis language model identifier, semantic encoding model identifier, prompt word version identifier, snapshot generation time, and snapshot status.
[0161] After all parameters and version information are successfully written, the next baseline snapshot is switched from the pending release state to the released state within the same database transaction, for binding by subsequent review batches. If the state switch fails, the release operation is rolled back, and the previous released baseline snapshot is used again. After the next baseline snapshot is released, the structured consistency verification report, review work order, and judgment result of this batch are still associated with the baseline version identifier bound to this batch.
[0162] To further aid in understanding the technical solutions of the embodiments of the present invention, a specific application example is given below.
[0163] This application example uses the review of a science and technology project on a 100-point scale. One review batch includes 5 experts and 12 secondary indicators. The AI sampling count is set to 5, the sampling temperature to 0.3, the minimum number of characters for comments to 8, the minimum number of valid experts to 3, the minimum deviation sequence length to 6, and the number of batches in the profile window to 5. The lower limit of the mapping gray area is set to 0.72, the mapping similarity limit to 0.85, and the minimum sample size to 30. The batch weight is set to 0.6, the scale offset limit to 0.08, the direction consistency ratio limit to 0.70, and the overcorrection upper limit to 0.15.
[0164] The high-score interval limit is set to 0.75, the low-score interval limit to 0.40, the bias threshold to 0.35, the lower limit of the bias confidence level to 0.55, the material support similarity limit to 0.78, and the numerical tolerance to 0.02. The initial bias threshold is set to 0.10, the lower limit of the threshold to 0.04, the initial decision confidence multiple to 2, the AI instability relaxation factor γ to 1.5, and the AI stability limit to 0.06. The initial discrete baseline is set to 0.12, the lower limit of the discrete baseline to 0.08, the lower limit of the mean to 0.15, the inter-cluster spacing limit to 0.25, and the minimum weight coverage to 0.80. The target false positive rate and the target false negative rate are both set to 10%, the adjustment step size to 0.1, and the lower and upper limits of the confidence multiple to 1 and 3, respectively.
[0165] The scores for the five AI inference outputs were 80, 82, 81, 83, and 84, respectively. The corresponding sampled score components were 0.80, 0.82, 0.81, 0.83, and 0.84, with a median of 0.82. Therefore, the AI score component is 0.82. The AI output dispersion is: ; The five experts' scores at this node were 0.74, 0.76, 0.75, 0.73, and 0.92, respectively. The fifth expert's in-batch offset was 0.10, historical offset was 0.08, and their scoring scale offset was: ; The expert's consistency rate in this batch is 0.80, the consistency rate in the historical direction is 0.75, and the consistency rate in the deviation direction is 0.78, which meets the scoring scale correction conditions. The corrected score component is 0.828.
[0166] After correction, the consensus distances of the five experts were 0.015, 0.015, 0, 0.025, and 0.083, respectively. The outlier threshold was 0.035, therefore the fifth expert was identified as an outlier. After removing this expert, the mean of robust experts was 0.745, and the AI expert indicator deviation was 0.075. The mean absolute value of the historical AI expert indicator deviation for this node was 0.035, the sample standard deviation was 0.012, and the dynamic deviation threshold was: ; The actual deviation judgment threshold is: ; The absolute value of the AI expert indicator deviation, 0.075, is greater than the actual deviation judgment threshold of 0.059, therefore this node is marked as a difference indicator. The AI output dispersion, 0.0158, is less than the AI stability limit of 0.06, and therefore does not constitute an AI output instability indicator.
[0167] The AI-generated sub-comments for this node, after parsing, yielded positive, neutral, and negative confidence scores of 0.18, 0.20, and 0.62, respectively, with a semantic tendency component of -0.44. Since the AI scoring component was 0.82, this node was marked as a self-consistency conflict indicator. The highest similarity between the factual statements in the comments and the candidate supporting segments was 0.63, lower than the material support similarity threshold of 0.78, thus resulting in a comment without material support.
[0168] See appendix Figure 9 To verify the effectiveness of the verification, 12 secondary indicators from 120 submitted projects were selected, resulting in 1440 node records. After manual review, 96 of these records were determined to be actual anomalies. A fixed deviation threshold of 0.10 was used for the comparison method. This embodiment employs the aforementioned scoring scale correction, dynamic threshold, AI output dispersion, and conflict attribution processing.
[0169] The comparative method identified 73 actual anomalies, generated 31 false positives, and missed 23 actual anomalies; this embodiment identified 84 actual anomalies, generated 18 false positives, and missed 12 actual anomalies. In this example, the proportion of identified actual anomalies to the total number of marked items is denoted as accuracy, and the proportion of identified actual anomalies to the total number of actual anomalies is denoted as recall. The calculated accuracy, recall, F1 score, measured false positive rate, and measured false negative rate of the comparative method are 70.2%, 76.0%, 73.0%, 29.8%, and 24.0%, respectively; the corresponding rates in this embodiment are 82.4%, 87.5%, 84.8%, 17.6%, and 12.5%. The results show that this embodiment can reduce false positives and missed anomalies.
[0170] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative examples. Those skilled in the art can omit, substitute, and change the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function in substantially the same way to achieve substantially the same result falls within the scope of the present invention.
Claims
1. A method for verifying the consistency of review results in artificial intelligence, characterized in that, Includes the following steps: Read the review form data, bind the baseline snapshot and generate a structured review dataset, obtain the expert rating components, perform independent inference, and obtain the AI output dispersion and AI rating components; The batch of indicator nodes are read from the structured review dataset and mapped to the unified logical indicator node. The expert's rating components and the AI rating components are then aggregated into the unified logical indicator node. The scoring scale offset is determined based on the expert's scoring components. The scoring scale is then corrected based on the scoring scale offset for experts who meet the preset correction conditions, resulting in the corrected scoring components. Outlier experts are identified and robust expert mean is calculated based on the corrected scoring components. AI expert index deviation is obtained based on the AI scoring components and robust expert mean. Dynamic deviation threshold used for this verification is obtained from the baseline snapshot. Actual deviation judgment threshold is obtained by combining the AI output dispersion. Difference index is marked based on AI expert index deviation and actual deviation judgment threshold to obtain the marked items. Attribution rules are applied to the tagged items to obtain the main attribution label and the additional attribution label; A structured consistency verification report and a review work order are generated based on the primary attribution label and the additional attribution label. The review confirmation result of the review work order is received, and the next baseline snapshot is generated based on the review confirmation result.
2. The method for verifying the consistency of artificial intelligence review results according to claim 1, characterized in that, The steps to obtain the corrected score components include: Using the node deviation formula, the deviation sequence of this batch is constructed according to the median of the score components of each expert relative to the other experts, to obtain the intra-batch offset and the proportion of directional consistency of this batch. The score scale profile is read, and the historical offset and the proportion of historical directional consistency are obtained from the valid historical deviation sequence in the score scale profile. When both the intra-batch offset and the historical offset are valid, the score scale offset is determined using the scale offset formula, and the deviation direction consistency ratio is obtained according to the proportion of directional consistency of this batch and the proportion of historical directional consistency. When the historical offset is empty, the ratio of the scoring scale offset to the deviation direction is taken as the ratio of the batch offset to the current batch direction. When the current batch deviation sequence length is less than the preset minimum deviation sequence length, the ratio of the scoring scale offset to the deviation direction is taken as the ratio of the historical offset to the historical direction. When the absolute value of the scoring scale offset is greater than the preset scale offset limit and the deviation direction consistency ratio is greater than the preset direction consistency ratio limit, the scoring scale offset is subtracted from the expert's scoring component. When the absolute value of the rating scale offset is greater than the preset overcorrection upper limit, if the rating scale offset is positive, the expert's rating component is subtracted from the overcorrection upper limit; if the offset is negative, the expert's rating component is added to the overcorrection upper limit to obtain the corrected rating component.
3. The method for verifying the consistency of review results in artificial intelligence according to claim 2, characterized in that, The steps for labeling difference indicators include: The number of effective experts is obtained by statistically aggregating the expert rating components to the unified logical indicator node. The consistency distance is obtained by the absolute value of the difference between the corrected rating component and the median of the corrected rating components of the other experts. The sum of the median of all consistency distances and the absolute deviation of twice the median is used as the outlier expert judgment boundary. The smaller value between the number of effective experts divided by 3 and rounded down and the number of effective experts minus the preset minimum number of effective experts is taken as the upper limit of the number of experts to be removed. Experts whose consistency distance is greater than the outlier expert judgment threshold are removed according to the upper limit of the removal quantity. The arithmetic mean of the corrected score components of the remaining experts is used as the robust expert mean. The AI score component is subtracted from the robust expert mean to obtain the AI expert index deviation. When the baseline snapshot has corresponding historical parameters, the dynamic deviation threshold used in this verification is calculated using a dynamic threshold formula. Otherwise, the dynamic deviation threshold used in this verification is taken as the preset initial deviation threshold. When the AI output dispersion is empty, the actual deviation judgment threshold is taken as the dynamic deviation threshold used in this verification. When the absolute value of the AI expert index deviation is greater than the actual deviation judgment threshold, it is marked as the difference index.
4. The method for verifying the consistency of artificial intelligence review results according to claim 3, characterized in that, The steps to map to a unified logical metric node include: The indicator hierarchy of the indicator nodes in this batch is read from the structured review dataset. A similarity matrix is generated under the constraint of the indicator hierarchy to obtain the similarity. Within the same batch, an indicator node in this batch is mapped to at most one unified logical indicator node. A unified logical indicator node accepts at most one indicator node in this batch. The mapping result is determined when the similarity is not less than the preset mapping similarity limit. When the similarity is not less than the preset lower limit of the mapping gray area and less than the mapping similarity limit, a mapping confirmation item is generated, and a temporary new unified logical indicator node is obtained. When the similarity is less than the lower limit of the mapped gray area, it is determined to be a newly added unified logical indicator node. The temporarily added unified logical indicator node and the newly added unified logical indicator node adopt the initial deviation threshold. The expert rating components, the valid rating components after excluding invalid ratings from the AI rating components, and the number of valid experts are aggregated into the corresponding unified logical indicator node. The number of valid historical samples and the average of historical rating components are obtained from the baseline snapshot. When the number of valid historical samples is not less than the preset minimum sample size, the similarity is not less than the mapping similarity limit, and the absolute value of the difference between the average of the historical rating components is not greater than the scale offset limit, a node merging item is generated. After confirmation, the valid historical samples are included in the confirmed merged unified logical index node.
5. The method for verifying the consistency of artificial intelligence review results according to claim 4, characterized in that, The steps to obtain the AI output dispersion and AI score components include: For the same application project and the same indicator node in this batch, the independent inference is performed according to the preset number of AI samplings. The same prompt words and the same decoding parameters except for the random state are used in each inference. The valid sub-item scores of each inference result are retained. The median of the valid sub-item scores is taken as the AI sub-item score. The AI sub-item score is converted into the AI score component. The AI output dispersion is obtained from the score component of the valid sub-item score. When the number of valid results is less than 2, the AI output dispersion is empty.
6. The method for verifying the consistency of artificial intelligence review results according to claim 5, characterized in that, The steps for obtaining the actual deviation judgment threshold by combining the AI output dispersion include: When the AI output dispersion is not empty, the actual threshold formula is used. The actual deviation judgment threshold is calculated based on the dynamic deviation threshold used in this verification, the preset AI instability relaxation coefficient and the AI output dispersion. When the AI output dispersion is greater than the preset AI stability limit, it is marked as an AI output instability index, and an AI re-review task is generated. The corresponding data is not included in the AI expert index deviation sample in the effective historical sample set.
7. The method for verifying the consistency of review results in artificial intelligence according to claim 6, characterized in that, The steps for generating the next baseline snapshot based on the verification results include: The structured consistency verification report is generated based on the main attribution label, the additional attribution label, and the marked item. The review priority score is obtained based on the marked item, and the review work order is generated according to the review priority score. Data that has not been marked are made into unmarked items. A first set of sampling review work orders and a second set of sampling review work orders are drawn from the marked items and the unmarked items respectively according to a preset sampling review ratio. The review confirmation results of the first set of sampling review work orders and the second set of sampling review work orders are received. The set of valid historical samples is updated according to the review confirmation results. When the number of valid historical samples is not less than the minimum sample size, the dynamic deviation threshold is recalculated using the dynamic threshold formula and written into the next baseline snapshot in the pending release state.
8. The method for verifying the consistency of artificial intelligence review results according to claim 7, characterized in that, The steps for generating the next baseline snapshot also include: The confidence multiple is determined from the baseline snapshot, the measured false positive rate is obtained from the first group of sampling review work orders, and the measured false negative rate is obtained from the second group of sampling review work orders and the unmarked items. When the measured false detection rate is greater than the preset target false detection rate and the measured false negative rate is not greater than the preset target false negative rate, the determination confidence factor is increased by a preset adjustment step size. When the measured false negative rate is greater than the target false negative rate and the measured false positive rate is not greater than the target false positive rate, the determination confidence factor is reduced by the adjustment step size. When the measured false positive rate is greater than the target false positive rate and the measured false negative rate is greater than the target false negative rate, an insufficient verification dimension prompt is generated and written into the structured consistency verification report. Once the preset termination conditions are met for this batch, the next baseline snapshot will be switched from the pending release state to the released state without rewriting the baseline snapshot.
9. The method for verifying the consistency of review results in artificial intelligence according to claim 8, characterized in that, The steps to obtain the primary attribution label and the secondary attribution label include: Based on the order of differences in scoring scale, AI applicability, indicator understanding, and unclassified differences, the first result that matches is determined as the primary attribution label, and the rest are the secondary attribution labels.
10. A consistency verification system for artificial intelligence review results, applied to the consistency verification method for artificial intelligence review results as described in any one of claims 1-9, characterized in that, include: The review data acquisition module is used to read review form data, bind baseline snapshots and generate structured review datasets, obtain expert rating components, perform independent inference, and obtain AI output dispersion and AI rating components. The indicator semantic alignment module is used to read the indicator nodes of this batch from the structured review dataset, map them to the unified logical indicator nodes, and aggregate the expert's rating components and the AI rating components to the unified logical indicator nodes. The scoring scale correction module is used to determine the scoring scale offset based on the expert's scoring components, and to perform scoring scale correction on experts who meet the preset correction conditions based on the scoring scale offset to obtain the corrected scoring components. The consistency verification module is used to identify outlier experts and calculate the mean of robust experts based on the corrected scoring components, obtain the AI expert index deviation based on the AI scoring components and the mean of robust experts, obtain the dynamic deviation threshold used for this verification from the baseline snapshot, obtain the actual deviation judgment threshold by combining the AI output dispersion, and mark the difference index based on the AI expert index deviation and the actual deviation judgment threshold to obtain the marked items. The conflict attribution analysis module is used to execute attribution rules on the marked items to obtain the main attribution label and the additional attribution label. The verification result generation module is used to generate a structured consistency verification report and a review work order based on the main attribution label and the additional attribution label, receive the review confirmation result of the review work order, and generate the next baseline snapshot based on the review confirmation result.
