An intelligent system evaluation method for data credibility of laboratory data recognition data

Abstract

This invention discloses an intelligent system evaluation method for identifying data credibility in laboratory data, relating to the fields of federated learning and data credibility evaluation. It solves the technical problems of difficulty in comprehensively considering model performance and uniqueness to improve the quality of global model aggregation, and the difficulty in designing differentiated bias calculations for different data collection scenarios and data types in laboratories. First, each experimental node trains a local credibility evaluation model using compliant data. The central server evaluates the model's contribution score based on a public benchmark dataset. Then, the central server aggregates the model parameter increments using a weighted federated average algorithm to generate a global consensus credibility model. New data is then jointly evaluated by the global and local models, calculating scenario-sensitive bias and recording distribution statistics. Finally, a lightweight prediction model is trained based on historical data, inputting various parameters such as scenario-sensitive bias to output the probability of exceeding a threshold, thus effectively determining data credibility.

Description

Technical Field

[0001] This invention belongs to the field of federated learning and data credibility assessment technology, specifically relating to an intelligent system assessment method for identifying data credibility in laboratory data. Background Technology

[0002] With the development of laboratory testing technology, the amount of data is growing exponentially, and the reliability of the data directly affects the accuracy of experimental results and the consistency of industry consensus.

[0003] Currently, the reliability assessment of laboratory data faces the following technical limitations: Traditional centralized assessment methods require the collection of raw data from each laboratory, posing a risk of data leakage; while decentralized assessments rely solely on local models, making it easy for local data bias to cause assessment results to deviate from industry benchmarks, thus failing to form cross-laboratory consensus; existing federated learning models often use data volume as the weight for aggregation, ignoring model performance and uniqueness, resulting in the underestimation of the contribution of high-performance models and the interference of low-quality models with the accuracy of the overall model; laboratory data types are diverse, and the collection scenarios vary greatly (e.g., routine testing, special environment testing, emergency calibration), and traditional fixed bias calculation methods do not take into account the characteristics of these scenarios.

[0004] The aforementioned issues mean that existing laboratory data credibility assessment methods cannot meet industry needs in terms of privacy protection, assessment accuracy, and scenario adaptability. There is an urgent need for an intelligent assessment method that takes into account privacy, consensus, performance, and predictive capabilities. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in the prior art; to this end, this invention proposes an intelligent system evaluation method for laboratory data identification data credibility, to solve the following technical problem:

[0006] Traditional centralized evaluation methods require the collection of raw data from each laboratory, posing a risk of data leakage. Decentralized evaluation, on the other hand, relies solely on local models, making it susceptible to biases in local data that could cause evaluation results to deviate from industry benchmarks and hinder cross-laboratory consensus. Existing federated learning models often use data volume as the weight for aggregation, ignoring model performance and uniqueness. This leads to the underestimation of the contribution of high-performance models and the interference of low-quality models with the accuracy of the overall model. Furthermore, laboratories have diverse data types and significantly different collection scenarios (e.g., routine testing, special environment testing, emergency calibration), and traditional fixed-bias calculation methods do not take these scenario characteristics into account.

[0007] To address the above problems, this invention provides an intelligent system evaluation method for laboratory data identification data credibility, comprising the following steps:

[0008] S1: Each laboratory node participating in federated learning trains a local credibility assessment model using its own compliant data. After receiving the models from each node, the central server evaluates the contribution score of each local credibility assessment model on a public benchmark dataset. The contribution score is obtained by weighted addition of the comprehensive performance evaluation score and the uniqueness evaluation score.

[0009] S2: The central server receives parameter increments from each laboratory model via an encrypted link, and aggregates them using a weighted federated average algorithm to generate a global consensus credibility model.

[0010] S3: Each laboratory node uses a global consensus credibility model and a local credibility evaluation model to evaluate the same batch of newly generated laboratory data, respectively, and obtains a global credibility score and a local credibility score. The absolute deviation between the global credibility score and the local credibility score is calculated, and the distribution statistics of the score difference between the two models on all data points are recorded.

[0011] S4: Each laboratory node trains a lightweight prediction model based on historical data from the past 3 months. The scene sensitivity bias, scene bias trend, and high bias sample features of the current sample are clustered and input into the trained model. The model outputs the probability of the bias exceeding the threshold. If the probability of the bias exceeding the threshold is greater than or equal to the preset probability threshold, an early warning is triggered; otherwise, the credibility assessment result of the current data is deemed valid.

[0012] The scene deviation trend is obtained by calculating the slope of the change of the average scene sensitivity deviation of each layer according to the collection time series; the average scene sensitivity deviation is statistically analyzed by layer, and high deviation samples with a scene sensitivity deviation ≥ 1.5 times the average scene sensitivity deviation are marked. The high deviation sample feature clustering is carried out by clustering the core features of the high deviation samples and statistically analyzing the feature category with the highest proportion.

[0013] Preferably, step S1 includes the following steps:

[0014] The public benchmark dataset consists of typical reliable and unreliable data samples from the laboratory industry.

[0015] The contribution score is specifically as follows:

[0016]

[0017] in, For the first The contribution score of each laboratory node. For the first The combined performance evaluation score of the local credibility assessment models on the public benchmark dataset. For the first The uniqueness assessment score of each local credibility assessment model and These are the corresponding weighting coefficients;

[0018] The uniqueness assessment score is obtained by defining preset data augmentation functions based on different types of laboratory data. After obtaining the relevant parameters of positive and negative samples using the preset data augmentation functions, the difference between the predicted distribution and the global average distribution is obtained by comparing the KL divergence quantification model.

[0019] The overall performance evaluation score is obtained through the first The accuracy, recall, precision, and mean squared error of a local credibility assessment model on a public benchmark dataset are obtained by weighted summation.

[0020] Preferably, defining preset data augmentation functions according to different types of laboratory data includes the following steps:

[0021] Define a preset data augmentation function, specifically for a public benchmark dataset. For different types of laboratory data, appropriate data augmentation strategies are adopted:

[0022] If the data is numerical detection data, the preset data augmentation function is:

[0023]

[0024] in, For preset data augmentation functions, The original samples of numerical detection data. For random noise that follows a normal distribution N(0, 0.05), The standard deviation is calculated for all numerical data of the same type as the original sample.

[0025] If the data is feature vector-based detection data, the preset data augmentation function is:

[0026]

[0027] in, For preset data augmentation functions, For feature masking operation function, The original samples of feature vector-based detection data. To conceal the proportions.

[0028] Preferably, obtaining the relevant parameters of positive and negative samples includes the following steps:

[0029] For each original sample in the public benchmark dataset ,in, ,in, Given the total number of samples in the public benchmark dataset, generate one corresponding negative sample using a pre-defined data augmentation function. To form positive and negative sample sets;

[0030] Input the positive and negative sample sets into all samples respectively. The local credibility assessment model of the first laboratory, where, for the first... A local credibility assessment model Output the set of predicted probability distributions for positive samples and the set of predicted probability distributions for negative samples, respectively.

[0031] At the same time, for each positive sample Calculate all The average value of the probability prediction distribution of each model is used to iterate through all positive samples to obtain the global average distribution set of positive samples; the same method is used to obtain the global average distribution set of negative samples.

[0032] Preferably, the uniqueness assessment score includes the following steps:

[0033] The difference between the distribution predicted by the KL divergence quantification model and the global mean distribution is compared, and the result is used as the uniqueness assessment score for calculation. Specifically: ;in, For the first The uniqueness assessment score of each local credibility assessment model The total number of samples in the public benchmark dataset. Let KL divergence be the KL divergence. and The first The local credibility assessment model for the first The predicted probability distribution of positive and negative samples. and For all models, the first... The average probability prediction distribution of positive and negative samples It is a regulating factor.

[0034] Preferably, step S2 uses a weighted federated average algorithm for aggregation, including the following steps:

[0035] The central server uses contribution scores as weights to perform a weighted average of the received local credibility assessment model parameter increments in order to calculate the initial global model update direction.

[0036] 10% of the samples in the public benchmark dataset are randomly selected as the consensus validation set;

[0037] A multi-objective optimization function is constructed by integrating accuracy, recall, and robustness performance in three dimensions;

[0038] The trust region constraint is set with the initial global model update direction as the center, specifically as follows:

[0039]

[0040] in, This indicates the direction for updating candidate global models. This sets the initial global model update direction. The radius of the trust domain;

[0041] The central server performs lightweight gradient descent locally to find the optimal update direction that minimizes the multi-objective optimization function.

[0042] The optimal update direction is superimposed on the current global model parameters, and the 3D performance is evaluated through a consensus verification set. Only when all thresholds are met is the data encrypted and distributed to each laboratory to form a global consensus credibility model.

[0043] Preferably, the multi-objective optimization function includes the following steps:

[0044] The multi-objective optimization function is obtained by weighted fusion of three-dimensional losses, specifically as follows:

[0045]

[0046] in, For a multi-objective optimization function, This indicates the direction for updating candidate global models. , and These represent losses in accuracy, recall, and robustness, respectively. , and These are the weighting coefficients corresponding to the accuracy, recall, and robustness losses, respectively.

[0047] Preferably, step S3 includes the following steps:

[0048] Each laboratory node divides newly generated laboratory data locally into different levels based on the collection scenario and labels each level with a scenario tag.

[0049] The global consensus credibility model and the local credibility evaluation model automatically call scenario-specific evaluation parameters for different scenario labels, and output the global credibility score set and the local credibility score set for each layer of data respectively;

[0050] The basic absolute deviation of a single sample in each layer is calculated, and a scenario deviation coefficient is introduced. The scenario deviation coefficient is determined by statistical analysis of historical laboratory data and specifically includes: the coefficient for routine testing scenarios, the coefficient for testing scenarios in special environments, and the coefficient for emergency calibration scenarios.

[0051] Preferably, the scene-sensitive deviation includes the following steps:

[0052] The scene-sensitive deviation is obtained by correcting the basic absolute deviation using a scene deviation coefficient.

[0053]

[0054] in, Due to scene-sensitive bias, This is the scene deviation coefficient. Basic absolute deviation;

[0055] After averaging the scene sensitivity bias by layer, high-bias samples with a scene sensitivity bias ≥ 1.5 times the average scene sensitivity bias are marked.

[0056] Preferably, step S4 includes the following steps:

[0057] The scene deviation trend is obtained by calculating the slope of the change in the average scene sensitivity deviation of each layer according to the collection time series.

[0058] The high-bias sample feature clustering involves clustering the core features of high-bias samples and statistically identifying the feature category with the highest percentage.

[0059] Each laboratory node trains a lightweight prediction model based on historical data from the past three months. The model then inputs the scene sensitivity bias, scene bias trend, and high-bias sample features of the current sample into the trained model and outputs the probability of the bias exceeding the threshold.

[0060] If the probability of deviation exceeding the threshold is greater than or equal to the preset probability threshold, an advance warning is triggered; otherwise, the current data credibility assessment result is deemed valid.

[0061] The beneficial effects of this invention are:

[0062] This invention trains models locally in each laboratory, and the central server uses a public benchmark dataset to construct a contribution score by integrating the comprehensive performance evaluation score and the uniqueness evaluation score. This avoids the underestimation of high-value models due to the dominance of data volume in the weighting, making the global model aggregation more accurate and forming a credible consensus across laboratories.

[0063] This invention achieves simultaneous optimization of the accuracy, recall, and robustness of the global model by first calculating the initial global model update direction with contribution scores as weights during aggregation, then constructing a multi-objective optimization function, using trust domain constraints and lightweight gradient descent to find the optimal direction, and finally distributing it after evaluation through a consensus validation set. This avoids single performance bottlenecks. At the same time, the trust domain constraint ensures that the optimal update direction does not deviate from the consensus of parameters in various laboratories, improves the compatibility between the global model and local models, and reduces subsequent evaluation bias.

[0064] This invention stratifies and labels new data according to the collection scenario, uses a dual model (local credibility assessment model and global consensus credibility model) to call scenario-specific parameters for evaluation, introduces scenario coefficients (routine detection and special environment) to calculate the scenario-sensitive bias, records the scenario bias trend and high-bias sample feature clustering statistics, and realizes multi-dimensional statistics to provide a basis for subsequent bias attribution.

[0065] This invention trains a lightweight prediction model based on three months of historical data. It takes scene sensitivity deviation, scene deviation trend and high deviation sample feature clustering as inputs and outputs the probability of exceeding a threshold. A corresponding probability threshold is preset. If the output probability of exceeding the threshold is greater than the preset probability threshold, an early warning is triggered. Otherwise, the data credibility assessment result is determined to be valid. This realizes advance prediction, allows the laboratory time to investigate problems, improves investigation efficiency and reduces invalid assessment results. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Please see Figure 1 As shown, this invention provides an intelligent system evaluation method for identifying the credibility of laboratory data, comprising the following steps:

[0069] S1: Each laboratory node participating in federated learning trains a local credibility assessment model using its own compliant data. After receiving the models from each node, the central server evaluates the contribution score of each local credibility assessment model on a public benchmark dataset. The contribution score is obtained by weighted addition of the comprehensive performance evaluation score and the uniqueness evaluation score.

[0070] S2: The central server receives parameter increments from each laboratory model via an encrypted link, and aggregates them using a weighted federated average algorithm to generate a global consensus credibility model.

[0071] S3: Each laboratory node uses a global consensus credibility model and a local credibility evaluation model to evaluate the same batch of newly generated laboratory data, respectively, and obtains a global credibility score and a local credibility score. The absolute deviation between the global credibility score and the local credibility score is calculated, and the distribution statistics of the score difference between the two models on all data points are recorded.

[0072] S4: Each laboratory node trains a lightweight prediction model based on historical data from the past 3 months. The scene sensitivity bias, scene bias trend, and high bias sample features of the current sample are clustered and input into the trained model. The model outputs the probability of the bias exceeding the threshold. If the probability of the bias exceeding the threshold is greater than or equal to the preset probability threshold, an early warning is triggered; otherwise, the credibility assessment result of the current data is deemed valid.

[0073] The scene deviation trend is obtained by calculating the slope of the change of the average scene sensitivity deviation of each layer according to the collection time series; the average scene sensitivity deviation is statistically analyzed by layer, and high deviation samples with a scene sensitivity deviation ≥ 1.5 times the average scene sensitivity deviation are marked. The high deviation sample feature clustering is carried out by clustering the core features of the high deviation samples and statistically analyzing the feature category with the highest proportion.

[0074] Specifically, each laboratory trains a local credibility assessment model using its own compliant data. After receiving the model, the central server calculates the comprehensive performance evaluation score and the uniqueness evaluation score of the model on a public benchmark dataset containing both credible and untrustworthy samples. These scores are then weighted and summed to obtain the contribution score. The central server receives the incremental model parameters from each laboratory via an SM4 encrypted link and calculates the initial update direction using a weighted average with the contribution score as the weight. A weighted federated average algorithm with multi-objective optimization (integrating accuracy, recall, and robustness loss) and trust domain constraints is used to find the optimal update direction through lightweight gradient descent. This direction is then superimposed on the current global model parameters to generate a global consensus credibility model. Each laboratory evaluates the same batch of new data using both the global and local models to obtain two types of credibility scores. The absolute bias is calculated, and a scene coefficient is introduced to correct for scene-sensitive bias. The trend of scene bias and the clustering distribution statistics of high-biased samples are recorded. Finally, the laboratory trains a lightweight logistic regression model using historical data from the past three months. The scene-sensitive bias, scene bias trend, and high-biased sample clustering coefficient are input, and the output exceeds the threshold probability. If the probability exceeds the preset threshold, an alert is triggered; otherwise, the evaluation result is deemed valid.

[0075] In one embodiment of the present invention, step S1 includes the following steps:

[0076] The public benchmark dataset consists of typical reliable and unreliable data samples from the laboratory industry.

[0077] The contribution score is specifically as follows:

[0078]

[0079] in, For the first The contribution score of each laboratory node. For the first The combined performance evaluation score of the local credibility assessment models on the public benchmark dataset. For the first The uniqueness assessment score of each local credibility assessment model and These are the corresponding weighting coefficients;

[0080] The uniqueness assessment score is obtained by defining preset data augmentation functions based on different types of laboratory data. After obtaining the relevant parameters of positive and negative samples using the preset data augmentation functions, the difference between the predicted distribution and the global average distribution is obtained by comparing the KL divergence quantification model.

[0081] The overall performance evaluation score is obtained through the first The accuracy, recall, precision, and mean squared error of a local credibility assessment model on a public benchmark dataset are obtained by weighted summation.

[0082] Specifically, typical data from the laboratory industry is collected, including reliable data verified by industry standards and explicitly labeled unreliable data, ensuring a balanced sample size for both types of data. Data preprocessing is performed, including cleaning the collected data, removing duplicate data, and correcting incorrectly labeled data. The data is then categorized by data type, including numerical detection data (e.g., reagent concentration values) and feature vector detection data (e.g., multi-dimensional combinations of concentration, temperature, and pH values), forming a public benchmark dataset of over 5,000 records, which is stored in an encrypted database on a central server. This public benchmark dataset is further divided into a test set and a validation set, which are then input into the local database of the i-th laboratory. The credibility assessment model calculates core metrics including accuracy (the ratio of correctly classified samples in the test set to the total number of samples in the test set), recall (the ratio of correctly identified untrustworthy samples in the test set to the actual number of untrustworthy samples in the test set), precision (the ratio of correctly identified trustworthy samples in the test set to the number of samples the model classifies as trustworthy), and mean squared error (the mean squared error between the model's predicted credibility score and the true credibility label of the sample, where trustworthy is 1 and untrustworthy is 0). Weights are assigned to accuracy, recall, precision, and mean squared error based on expert ratings, and these weighted values ​​are summed to obtain the overall performance evaluation score. Specifically:

[0083]

[0084] in, For the first The combined performance evaluation score of the local credibility assessment models on the public benchmark dataset. , , and The first The accuracy, recall, precision, and mean squared error of a local credibility assessment model.

[0085] In one embodiment of the present invention, defining preset data augmentation functions according to different types of laboratory data includes the following steps:

[0086] Define a preset data augmentation function, specifically for a public benchmark dataset. For different types of laboratory data, appropriate data augmentation strategies are adopted:

[0087] If the data is numerical detection data, the preset data augmentation function is:

[0088]

[0089] in, For preset data augmentation functions, The original samples of numerical detection data. For random noise that follows a normal distribution N(0, 0.05), The standard deviation is calculated for all numerical data of the same type as the original sample.

[0090] If the data is feature vector-based detection data, the preset data augmentation function is:

[0091]

[0092] in, For preset data augmentation functions, For feature masking operation function, The original samples of feature vector-based detection data. To conceal the proportions.

[0093] Specifically, The masking ratio (valued at 0.1, meaning 10% of non-critical feature dimensions are randomly masked; critical dimensions are specified by laboratory industry standards, such as the concentration of core indicators in medical testing data). Missing feature values ​​after masking are filled with the industry average of that dimension.

[0094] In one embodiment of the present invention, obtaining the correlation parameters between positive and negative samples includes the following steps:

[0095] For each original sample in the public benchmark dataset ,in, ,in, Given the total number of samples in the public benchmark dataset, generate one corresponding negative sample using a pre-defined data augmentation function. To form positive and negative sample sets;

[0096] Input the positive and negative sample sets into all samples respectively. The local credibility assessment model of the first laboratory, where, for the first... A local credibility assessment model Output the set of predicted probability distributions for positive samples and the set of predicted probability distributions for negative samples, respectively.

[0097] At the same time, for each positive sample Calculate all The average value of the probability prediction distribution of each model is used to iterate through all positive samples to obtain the global average distribution set of positive samples; the same method is used to obtain the global average distribution set of negative samples.

[0098] Specifically, for each original sample in the public benchmark dataset ,in, ,in, Given the total number of samples in the public benchmark dataset, generate one corresponding negative sample using a pre-defined data augmentation function. This is done to form a positive sample set and a negative sample set, with a one-to-one correspondence between the positive and negative samples; the positive sample set is the original public benchmark dataset; the local credibility assessment model's prediction output for the samples is a binary probability distribution of the credible and untrustworthy classes, and the positive and negative sample sets are respectively input into the entire dataset. The local credibility assessment models of each laboratory are denoted as follows: For the first A local credibility assessment model Input positive samples Then, output its predicted probability distribution, and iterate through all positive samples to obtain... The set of probability distributions for positive samples; similarly, the set of probability distributions for input negative samples. After that, I received The set of negative sample probability distributions; simultaneously, for each positive sample Calculate all The average value of the probability prediction distribution of each model is used to iterate through all positive samples to obtain the global average distribution set of positive samples; the same method is used to obtain the global average distribution set of negative samples.

[0099] In one embodiment of the present invention, the uniqueness assessment score includes the following steps:

[0100] The difference between the distribution predicted by the KL divergence quantification model and the global mean distribution is compared, and the result is used as the uniqueness assessment score for calculation. Specifically: ;in, For the first The uniqueness assessment score of each local credibility assessment model The total number of samples in the public benchmark dataset. Let KL divergence be the KL divergence. and The first The local credibility assessment model for the first The predicted probability distribution of positive and negative samples. and For all models, the first... The average probability prediction distribution of positive and negative samples This is the adjustment factor, with a value of 0.5.

[0101] Specifically, the value of the adjustment factor is determined comprehensively based on the technical objectives, data characteristics, and experimental verification results of the laboratory data credibility assessment. The local credibility assessment model is constructed based on an improved XGBoost model that combines feature fusion with an attention mechanism. This overcomes the limitation of traditional XGBoost models that rely solely on a single feature input. The specific construction steps are as follows: the input layer receives multi-dimensional features (numerical features / vector features) from the laboratory data; the feature attention layer introduces an attention weight matrix and calculates the attention weight of each feature; the feature fusion layer concatenates the attention-weighted features with the original features, and then... The connection layer performs feature fusion and outputs fused features. The improved XGBoost classification layer adopts a decision tree, which takes the fused features as input and outputs a binary probability distribution of the trustworthy and untrustworthy classes. The compliant data of each i-th laboratory is used as part of the training set to train the model. The Adam optimizer is used (learning rate 0.01, 100 iterations). Every 20 iterations, the model performance is evaluated using a validation set (accuracy ≥ 0.92, recall ≥ 0.90 is considered satisfactory). If the performance is not satisfactory, the number of decision trees (±2) or the learning rate (±0.02) is adjusted until the training meets the standards, thus obtaining the local trustworthiness evaluation model of the i-th laboratory.

[0102] In one embodiment of the present invention, step S2 employs a weighted federated average algorithm for aggregation, including the following steps:

[0103] The central server uses contribution scores as weights to perform a weighted average of the received local credibility assessment model parameter increments in order to calculate the initial global model update direction.

[0104] 10% of the samples in the public benchmark dataset are randomly selected as the consensus validation set;

[0105] A multi-objective optimization function is constructed by integrating accuracy, recall, and robustness performance in three dimensions;

[0106] The trust region constraint is set with the initial global model update direction as the center, specifically as follows:

[0107]

[0108] in, This indicates the direction for updating candidate global models. This sets the initial global model update direction. The radius of the trust domain;

[0109] The central server performs lightweight gradient descent locally to find the optimal update direction that minimizes the multi-objective optimization function.

[0110] The optimal update direction is superimposed on the current global model parameters, and the 3D performance is evaluated through a consensus verification set. Only when all thresholds are met is the data encrypted and distributed to each laboratory to form a global consensus credibility model.

[0111] Specifically, 10% of the samples are randomly selected from the public benchmark dataset as the consensus validation set. This set includes both trusted and untrusted samples, which are positive and negative samples, respectively. A multi-objective optimization function is constructed by fusing accuracy, recall, and robustness performance. A trust domain constraint is set centered on the initial global model update direction, specifically:

[0112]

[0113] in, As candidate update directions, This sets the initial global model update direction. The radius of the trust domain is 0.1*. ;

[0114] The central server performs lightweight gradient descent locally to find the optimal update direction that minimizes the multi-objective optimization function. The optimal update direction is then superimposed on the current global model parameters, and the 3D performance is evaluated through a consensus verification set. Only when all thresholds are met is the new model encrypted and distributed to each laboratory.

[0115] In one embodiment of the present invention, the multi-objective optimization function includes the following steps:

[0116] The multi-objective optimization function is obtained by weighted fusion of three-dimensional losses, specifically as follows:

[0117]

[0118] in, For a multi-objective optimization function, This indicates the direction for updating candidate global models. , and These represent losses in accuracy, recall, and robustness, respectively. , and These are the weighting coefficients corresponding to the accuracy, recall, and robustness losses, respectively.

[0119] Specifically, the core requirements for assessing the credibility of laboratory data are: first, ensuring the accurate identification of credible data; second, capturing unreliable data; and finally, guaranteeing robustness against interference. This is achieved by combining industry statistical data and setting weighting coefficients to meet these requirements. ,in, , and The corresponding values ​​are 0.4, 0.35, and 0.25, respectively. If the laboratory scenario is a special environmental detection scenario, such as a high-precision detection scenario, the robustness weight is increased to 0.3, and the accuracy weight is decreased to 0.35. The accuracy loss is calculated by inputting trusted positive samples from the previously constructed consensus verification set one by one into the temporary global model. The model outputs the confidence probability of each sample, with a value range of [0,1]. For example, if a certain concentration sample outputs a confidence probability of 0.97, it means that the model determines its confidence probability to be 97%. The cross-entropy loss is used to quantify the deviation between the predicted probability and the true label to obtain the accuracy loss. The calculation formula is as follows: ;in, For the loss of accuracy, The total number of credible positive samples. This represents the true label of a positive sample, with a value of 1. The smaller the loss value, the higher the accuracy of the model in identifying reliable data.

[0120] The recall loss is calculated by inputting untrusted samples from the consensus verification set into a temporary global model, and the model outputs the confidence probability of each sample. Similarly, the cross-entropy loss method is used to obtain the recall loss.

[0121] Robustness loss calculation involves generating perturbation samples by adding adapted Gaussian noise to each reliable sample in the reliable positive sample set, according to the data type: Numerical samples: First, the standard deviation of all numerical data of the same type as the current sample is calculated. A Gaussian noise value is then superimposed on the original sample's numerical value. The distribution characteristics of this Gaussian noise are bound to the data type: the mean of the noise is set to 0 (ensuring no systematic bias in the noise, and preventing it from intentionally increasing or decreasing the original data), and the variance of the noise is set to 3% of the square of the standard deviation of that data type (i.e., the noise amplitude is strictly controlled within 3% of the natural fluctuation range of the data, avoiding excessive noise causing the perturbation sample to deviate from the true data distribution); Feature vector samples: First, according to laboratory industry standards, the multiple feature dimensions of the sample are divided into key dimensions. Non-critical dimensions – critical dimensions refer to core indicators that directly determine data reliability (such as concentration and temperature, the accuracy of which directly affects experimental results), while non-critical dimensions refer to auxiliary indicators (such as collection time and equipment number, the fluctuation of which has minimal impact on data reliability). Gaussian noise is added only to non-critical dimensions (the noise setting logic is consistent with that of numerical samples, i.e., the variance is 3% of the square of the standard deviation of the non-critical dimension data), while the original values ​​of critical dimensions remain unchanged to avoid the loss of evaluation significance of perturbed samples due to interference with core information. The predicted distribution of the original samples and the predicted distribution of the perturbed samples by the temporary global model are obtained separately, and the difference between the two is quantified by JS divergence. The average of the JS divergence of all samples of the reliable positive samples is taken as the robustness loss.

[0122] In one embodiment of the present invention, step S3 includes the following steps:

[0123] Each laboratory node divides newly generated laboratory data locally into different levels based on the collection scenario and labels each level with a scenario tag.

[0124] The global consensus credibility model and the local credibility assessment model automatically call scenario-specific assessment parameters for different scenario labels, and output the global credibility score set and the local credibility score set for each layer of data respectively;

[0125] The baseline absolute deviation of a single sample in each layer is calculated, and a scene deviation coefficient is introduced to correct the baseline absolute deviation to obtain the scene-sensitive deviation. The scene deviation coefficient is determined by statistical analysis of historical laboratory data and specifically includes: the coefficient for routine testing scenarios, the coefficient for testing scenarios in special environments, and the coefficient for emergency calibration scenarios.

[0126] Specifically, each laboratory node, based on the three-dimensional indicators of environmental conditions, task priority, and equipment status at the time of data acquisition, divides newly generated laboratory data (such as batch reagent testing data, part size measurement data) into three levels. The specific division rules are as follows:

[0127] Typical testing scenarios: Stable environmental conditions (e.g., temperature 25±2℃, humidity 50±5%), routine periodic testing tasks (e.g., daily reagent purity verification), and equipment in normal calibration status (calibration time ≤ 7 days from now).

[0128] Special environmental testing scenarios: environmental conditions deviate from the standard range (e.g., high temperature above 40℃, low temperature below -5℃, or humidity > 65%), special tests under special working conditions (e.g., material stability testing under high temperature environment), and equipment that has undergone temporary calibration (calibration time ≤ 24 hours from the current time, adapted to special environments).

[0129] Emergency calibration scenario: The task is to perform emergency verification after equipment malfunction (e.g., equipment accuracy verification when detection data suddenly fluctuates), the acquisition time is ≤30 minutes (results need to be obtained quickly), and the equipment is in a state of waiting for formal calibration (for temporary emergency use only).

[0130] The laboratory data management system automatically reads the acquisition logs of each new data entry (environmental sensor data, task type identifier, equipment calibration record), matches the scene level according to the above rules, and marks each data entry with a scene tag: Tag1 for routine testing scenarios, Tag2 for special environment testing scenarios, and Tag3 for emergency calibration scenarios.

[0131] Among them, the scenario-specific evaluation parameters are extracted from three preset scenario-specific evaluation parameters by the central server and each laboratory node. The parameters are customized for the core needs of different scenarios, specifically as follows:

[0132] Tag1 exclusive evaluation parameters: The model's tolerance threshold for abnormal data features is set to 0.1 (data fluctuations are small in normal scenarios, and anomalies are strictly identified), and the core indicators (such as reagent concentration) account for 0.7% of the feature weights (evaluation is based on core indicators first).

[0133] Tag2-specific evaluation parameters: The tolerance threshold for abnormal features is set to 0.3 (to avoid misjudgment of reasonable differences due to large fluctuations in data in special environments), the weight of core indicators is reduced to 0.5, and the weight of environmental adaptability indicators is increased to 0.3;

[0134] Tag3 exclusive evaluation parameters: Anomaly feature tolerance threshold set to 0.2 (balancing urgency and accuracy), equipment temporary calibration deviation correction coefficient set to 1.1 (compensating for minor errors in equipment under emergency conditions).

[0135] The laboratory node will input new data labeled with scene tags into the global consensus credibility model and the local credibility assessment model respectively. When the model receives the data, it will automatically read the scene tags and call the exclusive assessment parameters for the corresponding scene. For example, when inputting data labeled with Tag2, both models will automatically load special environmental detection parameters to compensate for the environmental fluctuation characteristics in the data. After the assessment is completed, both models will output the credibility score set for each layer of data.

[0136] In one embodiment of the present invention, the scene-sensitive deviation includes the following steps:

[0137] The scene-sensitive deviation is obtained by correcting the basic absolute deviation using a scene deviation coefficient.

[0138]

[0139] in, Due to scene-sensitive bias, This is the scene deviation coefficient. Basic absolute deviation;

[0140] After averaging the scene sensitivity bias by layer, high-bias samples with a scene sensitivity bias ≥ 1.5 times the average scene sensitivity bias are marked.

[0141] Specifically, for each sample under the same scene label, the basic absolute deviation between the global confidence score and the local confidence score is calculated. The formula is "Basic Absolute Deviation = |Global Score - Local Score|". Each laboratory node determines the scene deviation coefficient through statistical analysis based on historical data from the past 3 months (including deviation records for each scene and evaluation error events caused by the deviation). The specific rules are as follows:

[0142] Routine detection scenario coefficient: In historical data, the reasonable deviation ratio (the proportion of deviations that did not lead to evaluation errors) of the Tag1 scenario is 95%, and the average deviation is small. Therefore, the routine detection scenario coefficient is set to 1.0 (no need to amplify the deviation, directly reflecting the real difference).

[0143] Special environment detection scenario coefficient: The risk of deviation amplification due to environmental fluctuations in the Tag2 scenario (small deviations can easily lead to misjudgment in assessment) is 20% higher than that in Tag1. Therefore, the special environment detection scenario coefficient value is set to 1.2 (to moderately amplify the deviation and strengthen the focus on deviations in special scenarios).

[0144] Emergency calibration scenario coefficient: Due to the emergency state of the equipment, the tolerance margin for deviation in the Tag3 scenario is 10% lower than that in Tag1. Therefore, the emergency calibration scenario coefficient is set to 0.9 (to appropriately reduce the deviation and avoid oversensitivity that could lead to interruption of the emergency task).

[0145] The scene-sensitive deviation is obtained by correcting the basic absolute deviation using a scene deviation coefficient. The formula logic is: Scene-sensitive deviation = Scene deviation coefficient × Basic absolute deviation. The corrected scene-sensitive deviation can more accurately reflect the actual impact of deviation on the evaluation results under different scenarios, and provide a basis for scene adaptation for subsequent anomaly judgment.

[0146] In one embodiment of the present invention, step S4 includes the following steps:

[0147] The scene deviation trend is obtained by calculating the slope of the change in the average scene sensitivity deviation of each layer according to the collection time series.

[0148] The high-bias sample feature clustering involves clustering the core features of high-bias samples and statistically identifying the feature category with the highest percentage.

[0149] Each laboratory node trains a lightweight prediction model based on historical data from the past three months. The model then inputs the scene sensitivity bias, scene bias trend, and high-bias sample features of the current sample into the trained model and outputs the probability of the bias exceeding the threshold.

[0150] If the probability of deviation exceeding the threshold is greater than or equal to the preset probability threshold, an advance warning is triggered; otherwise, the current data credibility assessment result is deemed valid.

[0151] Specifically, based on the new data processed by each laboratory node (after scene stratification and scene sensitivity deviation calculation), a set of distribution statistics is constructed, including scene sensitivity deviation, scene deviation trend, and high-deviation sample feature clustering. For each scene data point, the high-deviation sample threshold is set to 1.5 times the average of the scene sensitivity deviations of all samples in that scene. The lightweight prediction model collects historical data from the laboratory over the past three months, labeling each data point with three types of input features and one type of label (input features: scene sensitivity deviation of historical data, scene deviation trend (encoded as numerical values: rising = 2, stable = 1, falling = 0) and "high-deviation core feature category (encoded as numerical values: e.g., Device-03 +)". Part diameter = 3, other categories are coded by serial number); Label: whether the historical data ultimately "exceeded the threshold" (1 = exceeded the threshold, 0 = did not exceed the threshold, labeled based on the actual evaluation results at that time)); A lightweight prediction model is constructed using "logistic regression combined with a lightweight fully connected layer". Input layer: 3 neurons (corresponding to 3 types of input features), hidden layer: 1, containing 8 neurons (activation function is ReLU, to reduce overfitting), output layer: 1 neuron (activation function is Sigmoid, outputting the probability of deviation exceeding the threshold, with a value range of [0,1]), using Adam optimizer (learning rate 0.01) and cross-entropy. The loss function is used for 50 training iterations. In each iteration, the model accuracy is evaluated using a validation set (≥85% is required; if it fails to meet the target, the number of hidden layer neurons is adjusted to 12, and retraining is performed). Finally, the trained model that meets the target is saved. The preset probability threshold is obtained by taking historical data from the laboratory over the past 3 months, filtering samples with bias exceeding the threshold probability minus the actual result, plotting the ROC curve (AUC≥0.85), setting the target recall rate according to the scenario (e.g., medical ≥95%, industrial ≥85%), and finding the critical point corresponding to the ROC curve as the initial preset probability threshold. After validation set testing and expert fine-tuning, the data is updated monthly with new data to adapt to the risk requirements of the scenario.

[0152] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. An intelligent system evaluation method for laboratory data recognition data credibility, characterized in that, Includes the following steps: S1: Each laboratory node participating in federated learning trains a local credibility assessment model using its own compliant data. After receiving the models from each node, the central server evaluates the contribution score of each local credibility assessment model on a public benchmark dataset. The contribution score is obtained by weighted addition of the comprehensive performance evaluation score and the uniqueness evaluation score. S2: The central server receives parameter increments from each laboratory model via an encrypted link, and aggregates them using a weighted federated average algorithm to generate a global consensus credibility model. In step S2, a weighted federated average algorithm is used for aggregation, which includes the following steps: The central server uses contribution scores as weights to perform a weighted average of the received local credibility assessment model parameter increments in order to calculate the initial global model update direction. 10% of the samples in the public benchmark dataset are randomly selected as the consensus validation set; A multi-objective optimization function is constructed by integrating accuracy, recall, and robustness performance in three dimensions; The trust region constraint is set with the initial global model update direction as the center, specifically as follows: wherein, is a candidate global model update direction, is an initial global model update direction, is a trust region radius; The central server performs lightweight gradient descent locally to find the optimal update direction that minimizes the multi-objective optimization function. The optimal update direction is superimposed on the current global model parameters, and the 3D performance is evaluated through a consensus verification set. Only when all thresholds are met is the data encrypted and distributed to each laboratory to form a global consensus credibility model. S3: Each laboratory node uses a global consensus credibility model and a local credibility evaluation model to evaluate the same batch of newly generated laboratory data, respectively, and obtains a global credibility score and a local credibility score. The absolute deviation between the global credibility score and the local credibility score is calculated, and the distribution statistics of the score difference between the two models on all data points are recorded. Step S3 includes the following steps: Each laboratory node divides newly generated laboratory data locally into different levels based on the collection scenario and labels each level with a scenario tag. The global consensus credibility model and the local credibility evaluation model automatically call scenario-specific evaluation parameters for different scenario labels, and output the global credibility score set and the local credibility score set for each layer of data respectively; The baseline absolute deviation of a single sample in each layer is calculated, and a scene deviation coefficient is introduced to correct the baseline absolute deviation to obtain the scene-sensitive deviation. The scene deviation coefficient is determined through statistical analysis of historical laboratory data and specifically includes: the coefficient for routine testing scenarios, the coefficient for testing scenarios in special environments, and the coefficient for emergency calibration scenarios. S4: Each laboratory node trains a lightweight prediction model based on historical data from the past 3 months. The scene sensitivity bias, scene bias trend, and high bias sample features of the current sample are clustered and input into the trained model. The model outputs the probability of the bias exceeding the threshold. If the probability of the bias exceeding the threshold is greater than or equal to the preset probability threshold, an early warning is triggered; otherwise, the credibility assessment result of the current data is deemed valid. The scene deviation trend is obtained by calculating the slope of the change of the average scene sensitivity deviation of each layer according to the collection time series; after statistically analyzing the average scene sensitivity deviation of each layer, high deviation samples with a scene sensitivity deviation ≥ 1.5 times the average scene sensitivity deviation are marked. The high deviation sample feature clustering is performed by clustering the core features of the high deviation samples and statistically analyzing the feature category with the highest proportion.

2. The intelligent system evaluation method for laboratory data identification data credibility according to claim 1, characterized in that, Step S1 includes the following steps: The public benchmark dataset consists of typical reliable and unreliable data samples from the laboratory industry. The contribution score is specifically as follows: in, For the first The contribution score of each laboratory node For the first The combined performance evaluation score of the local credibility assessment models on the public benchmark dataset. For the first The uniqueness assessment score of each local credibility assessment model and These are the corresponding weighting coefficients; The uniqueness assessment score is obtained by defining preset data augmentation functions based on different types of laboratory data. After obtaining the relevant parameters of positive and negative samples using the preset data augmentation functions, the difference between the predicted distribution and the global average distribution is obtained by comparing the KL divergence quantification model. The overall performance evaluation score is obtained through the first The accuracy, recall, precision, and mean squared error of a local credibility assessment model on a public benchmark dataset are obtained by weighted summation.

3. The intelligent system evaluation method for laboratory data identification data credibility according to claim 2, characterized in that, The process of defining preset data augmentation functions based on different types of laboratory data includes the following steps: Define a preset data augmentation function, specifically for a public benchmark dataset. Different types of laboratory data, among which, Given the total number of samples in the public benchmark dataset, an appropriate data augmentation strategy is adopted: If the data is numerical detection data, the preset data augmentation function is: in, For the preset data augmentation function, The original samples of numerical detection data. For random noise that follows a normal distribution N(0, 0.05), The standard deviation is calculated for all numerical data of the same type as the original sample. If the data is feature vector-based detection data, then the preset data augmentation function is: in, For the preset data augmentation function, For feature masking operation function, The original samples of feature vector-based detection data. To conceal the proportions.

4. The intelligent system evaluation method for laboratory data identification data credibility according to claim 2, characterized in that, The process of obtaining the relevant parameters of positive and negative samples includes the following steps: For each original sample in the public benchmark dataset ,in, One corresponding negative sample is generated using a preset data augmentation function. To form positive and negative sample sets; Input the positive and negative sample sets into all samples respectively. The local credibility assessment model of the first laboratory, where, for the first... A local credibility assessment model Output the set of predicted probability distributions for positive samples and the set of predicted probability distributions for negative samples, respectively. At the same time, for each positive sample Calculate all The average value of the probability prediction distribution of each model is used to iterate through all positive samples to obtain the global average distribution set of positive samples; the same method is used to obtain the global average distribution set of negative samples.

5. The intelligent system evaluation method for laboratory data identification data credibility according to claim 2, characterized in that, The uniqueness assessment score includes the following steps: The difference between the distribution predicted by the KL divergence quantification model and the global mean distribution is compared, and the result is used as the uniqueness assessment score for calculation. Specifically: ;in, For the first The uniqueness assessment score of each local credibility assessment model The total number of samples in the public benchmark dataset. Let KL divergence be the KL divergence. and The first The local credibility assessment model for the first The predicted probability distribution of positive and negative samples. and For all models, the first... The average probability prediction distribution of positive and negative samples It is a regulating factor.

6. The intelligent system evaluation method for laboratory data identification data credibility according to claim 1, characterized in that, The multi-objective optimization function includes the following steps: The multi-objective optimization function is obtained by weighted fusion of three-dimensional losses, specifically as follows: in, For a multi-objective optimization function, This indicates the direction for updating candidate global models. , and These represent losses in accuracy, recall, and robustness, respectively. , and These are the weighting coefficients corresponding to the accuracy, recall, and robustness losses, respectively.

7. The intelligent system evaluation method for laboratory data identification data credibility according to claim 1, characterized in that, The scenario-sensitive deviation includes the following steps: The scene-sensitive deviation is obtained by correcting the basic absolute deviation using a scene deviation coefficient. in, Due to scene-sensitive bias, This is the scene deviation coefficient. Based on absolute deviation.

Images