Abnormal data processing method and system for laboratory capability verification

Abstract

The invention relates to the technical field of data processing, in particular to an abnormal data processing method and system for laboratory capability verification. According to the method, a brand-new data preprocessing method is designed during proficiency verification and evaluation, that is, in the data filling operation process, the types of data to be filled are classified and judged firstly, then different data filling methods are selected in a targeted mode according to different types of data, and the data filling accuracy is improved; and meanwhile, aiming at the condition that the data point clock needs to be filled with more data, the data point with the highest correlation is searched as a reference to carry out data filling operation, so that the data filling accuracy can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an abnormal data processing method and system for laboratory capability verification. Background Art

[0002] As an important external quality evaluation activity, proficiency testing is an effective means of evaluating laboratory technical capabilities and an important part of laboratory quality control. As a general means of measuring quality, proficiency testing has also received more and more attention from the industry and academia. The traditional process of conducting proficiency testing activities, including invitation to register, sample preparation, sample distribution, result collection, data aggregation, etc., is basically completed manually by the organizer. Not only is the workload large and requires more human resources, but the operation is cumbersome and prone to errors. With the continuous advancement of science and technology, the manual organization of proficiency testing mode is no longer suitable for the needs of today's laboratory information development. Laboratory Information Management System (LIMS) is a new application technology that integrates modern management ideas and computer technology for laboratory information management and control in various industries. With the continuous development of information technology and food testing industry, the number of food testing institutions that need to participate in proficiency testing plans is increasing. In order to respond to the trend of information office and paperless office, the construction of software systems for unified and standardized management and the promotion of information construction of proficiency testing work are the research paths that all proficiency testing providers (PTPs) are exploring.

[0003] In the prior art, when evaluating laboratory proficiency testing, researchers mainly focus on what evaluation method to use to achieve proficiency testing evaluation, but pay less attention to the data processing process of proficiency testing evaluation, resulting in low accuracy of data used for proficiency testing evaluation and thus affecting the conclusion of proficiency testing evaluation. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an abnormal data processing method and system for laboratory proficiency verification, which improves the accuracy of proficiency verification evaluation by improving the proficiency verification data processing process.

[0005] The present invention provides a method for processing abnormal data for laboratory capability verification, comprising the following steps: S1: Obtain information of laboratories participating in proficiency testing.

[0006] S2: performing a preprocessing operation on the laboratory test data in the laboratory information; The pre-processing operation includes: S2.1: Detect abnormal data points on the laboratory test data; S2.2: Delete the detected abnormal data points; S2.3: Fill in missing values ​​in the test data; The S2.3 is specifically: S2.3.1: Classify and judge the missing values. If the number of missing values ​​in the data point is greater than a preset value, proceed to S2.3.2; otherwise, proceed to S2.3.3; S2.3.2: Use the adjacent value averaging method to perform data filling operation on the missing values; S2.3.3: Use the relevant data filling method to fill in the missing values; S3: Conduct laboratory proficiency testing evaluation based on the data preprocessed by S2.

[0007] Preferably, in S2.3.2, the formula for performing data filling operation on the missing value by the adjacent value averaging method is:

[0008] In the formula, x i is a missing value, x i-1 and x i+1 is the value of the missing value.

[0009] Preferably, the data filling operation for the missing values ​​using the relevant data filling method is specifically as follows: S2.3.3.1: Normalize all data points, and then calculate the distance between the data point to be filled and other data points; S2.3.3.2: Sort the distances to obtain k other data points with the smallest distances; The specific value determination formula of k is:

[0010] In the formula, is the average of the five minimum distances, a is the number of missing values ​​in the data points to be filled, and a n is the number of data in each data point, b is the number of data points, and r is the coefficient; S2.3.3.3: Fill in the missing values ​​in the data points to be filled based on the k other data points with the smallest distances.

[0011] Preferably, the Euclidean distance formula is used to calculate the data point x to be filled m With the complete data point x c distance; The Euclidean distance formula is specifically:

[0012] In the formula, X miis the normalized ith data point in the data point to be filled, i=1-n, X ci is the normalized ith data among the other data points corresponding to the position of the data point to be filled, and n is the number of data in the data point to be filled.

[0013] Preferably, in S2.3.1, the preset value is 1.

[0014] Preferably, in S2.1, abnormal data points are detected based on a k-means clustering algorithm.

[0015] Preferably, the S2.1 is specifically: S2.1.1: Calculate for each data point x in the laboratory test data i With the initial cluster center The distance between i ; S2.1.2: Determine the minimum distance between the test data point and the cluster center , according to the Assigning data points in the test data to corresponding clusters; S2.1.3: updating the cluster center; S2.1.4: Repeat S2.1.1-S2.1.3 until the cluster center and the objective function value remain unchanged; S2.1.5: Detect abnormal data points based on clustering results.

[0016] Preferably, in S1, the laboratory information includes: laboratory quantity and laboratory test data.

[0017] Preferably, in S2.1.1, each data point is associated with the initial cluster center The distance between is calculated as:

[0018] In the formula, x ij is the j-th test result of the i-th data point in the test data, is the jth dimension of the lth cluster center, and m is the number of data in each data point in the test data.

[0019] According to another aspect of the present invention, there is provided an abnormal data processing system for laboratory proficiency verification, the system adopts the above-mentioned abnormal data processing method for laboratory proficiency verification, and the system comprises: The information acquisition module is used to obtain the information of laboratories participating in the proficiency testing.

[0020] A data preprocessing module, used for performing preprocessing operations on the laboratory test data in the laboratory information; The capability verification module is used to perform laboratory capability verification evaluation based on the data preprocessed by the data preprocessing module.

[0021] The embodiments of the present invention have the following technical effects: The present invention designs a new data preprocessing method when evaluating capability verification, that is, in the process of data filling operation, the data type to be filled is first classified and judged, and then different data filling methods are selected according to different types of data, thereby improving the accuracy of data filling; at the same time, in the case where there are many data points to be filled, the operation of finding the data point with the highest correlation as a benchmark for data filling can effectively improve the accuracy of data filling; The present invention proposes a new formula for determining the specific value of k in a clustering algorithm, which improves the targeted filling of missing data in data points in laboratory proficiency testing, can achieve targeted filling of missing data in various types of data points, and improves the data filling accuracy of the data points to be filled. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a flow chart of an abnormal data processing method for laboratory capability verification provided by Example 1 of the present invention; Figure 2 This is a flowchart of the preprocessing operation provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0025] Embodiment 1, Figure 11 is a flow chart of the laboratory proficiency testing evaluation method provided in Example 1 of the present invention. Figure 1 , specifically including the following steps: S1: Obtain information of laboratories participating in proficiency testing.

[0026] Proficiency verification refers to a laboratory or an inspection agency participating in a plan issued by a plan provider as a plan participant, testing samples to obtain test results, and evaluating the capabilities of plan participants based on the comparison of test results between laboratories. In a broad sense, proficiency verification includes one-to-many and one-to-one proficiency verification, mainly including: laboratory proficiency verification plan, measurement audit plan. Laboratory proficiency verification plan refers to at least six laboratories testing the same sample. Measurement audit plan refers to a laboratory performing actual measurements on samples with determined specified values ​​and capability assessment standard deviations. Furthermore, the laboratory measurement audit plan will preset the specified values ​​and standard deviations of this plan before participating in the actual measurement of the samples. The laboratory will perform actual measurements on the samples to obtain the specified values ​​and standard deviations of the current measurements, and calculate the z-ratio score based on the determined specified values ​​and standard deviations. This embodiment will send an evaluation of whether the results of this measurement are satisfactory to the laboratory. If the laboratory is not satisfied with the current measurement results, it can continue to retest. Before each retest, this implementation needs to reset the preset values ​​of the specified values ​​and standard deviations until the laboratory measurement results are satisfactory. This embodiment is aimed at testing food, mainly for the determination of chemical properties of food, specifically the determination of chemical components in food, the determination of additives in food, the determination of nutritional components in food, etc., such as: the determination of non-volatile acid in vinegar, the determination of vitamin B6, taurine, choline in infant formula milk powder, the determination of chromium and arsenic in rice, etc.; this embodiment also includes a small amount of determination of non-chemical properties of food, such as the determination of moisture content of food, the determination of optical rotation of food, etc.; In this embodiment, the laboratory information includes: the number of laboratories and laboratory test data; wherein the laboratory test data is that each laboratory performs multiple tests on the test items, that is, each test item is a data point, and the data point includes multiple data; S2: performing a preprocessing operation on the laboratory test data in the laboratory information; Laboratory proficiency verification is relatively complex. One of the main steps is to judge the laboratory's inspection capability based on laboratory test data. This is based on the statistical analysis of specific data, which will directly affect the final evaluation of the inspection. In the process of data collection of laboratory information, since the noise of the test process and data recording process cannot be avoided, this will cause the laboratory test data to generate data that is inconsistent with the real time, that is, the appearance of abnormal data points. Therefore, it is necessary to pre-process the experimental information to eliminate abnormal data points; In this embodiment, if Figure 2 As shown, the preprocessing operation includes: S2.1: Detect abnormal data points on the laboratory test data; In the prior art, when performing anomaly detection on laboratory test data, data processing personnel generally screen and detect abnormal data points based on their own experience. However, the above method is greatly affected by the experience of data processing personnel in detecting abnormal data points, and the screening and detection quality is prone to be unstable. Therefore, this embodiment uses a k-means clustering algorithm for abnormal data point detection; Specifically, the S2.1 is as follows: S2.1.1: Calculate for each data point x in the laboratory test data i With the initial cluster center The distance between i ; Among them, each data point and the initial cluster center The distance between is calculated as:

[0027] In the formula, x ij is the j-th test result of the i-th data point in the test data, is the jth dimension of the lth cluster center, and m is the number of data in each data point in the test data; S2.1.2: Determine the minimum distance between the test data point and the cluster center , according to the Assigning data points in the test data to corresponding clusters; S2.1.3: updating the cluster center; Specifically, the formula for updating the cluster center is:

[0028] In the formula, is the updated cluster center, is the number of data points in the lth cluster, E is the objective function value of the test data, and n is the number of data points in the test data; S2.1.4: Repeat S2.1.1-S2.1.3 until the cluster center and the objective function value remain unchanged; S2.1.5: Detect abnormal data points based on clustering results; S2.2: Delete the detected abnormal data points; Through the above steps, the abnormal data points in the test data can be identified by the algorithm, and after the abnormal data points are detected, the abnormal data points are deleted; S2.3: Fill in missing values ​​in the test data; In fact, in the process of laboratory proficiency verification, the occurrence of outlier missing values ​​is relatively common, and there are many studies on the filling of missing values ​​in the existing technology. Generally, the adjacent value average method or the average value method is used for data filling operation; wherein, the adjacent value average method is for the situation where there are adjacent data before and after the missing position, and the average is calculated by using these adjacent values ​​to complete the difference operation. This method is based on the assumption of local continuity of data, that is, a data point is usually associated with its adjacent data points, and the difference between the two will be too large; the average value rule is to use the average value of the data distribution to fill the missing data, which is a kind of statistical value interpolation method. The purpose of this method is to reduce the impact of interpolated data on the overall data distribution, because most of the data in the real laboratory test environment obeys the normal distribution, that is, most of the test data are distributed around the average value; the above two methods are widely used in the process of data filling operation, but in the process of data filling operation, a method is used to fill the data, for example, the adjacent value average method or the average value method is used for data filling operation, that is, the actual situation of the missing data is not judged, which leads to reduced accuracy of data filling; In this step, S2.3 is specifically: S2.3.1: Classify and judge the missing values. If the number of missing values ​​in the data point is greater than a preset value, proceed to S2.3.2; otherwise, proceed to S2.3.3; Wherein, in this embodiment, the preset value is 1; S2.3.2: Use the adjacent value averaging method to perform data filling operation on the missing values; The formula for performing data filling operation on the missing values ​​by the adjacent value averaging method is:

[0029] In the formula, x i is a missing value, x i-1 and x i+1 is the value of the missing value; S2.3.3: Use the relevant data filling method to fill in the missing values; In this step, the data filling operation for the missing values ​​using the relevant data filling method is specifically as follows: S2.3.3.1: Normalize all data points, and then calculate the distance between the data point to be filled and other data points; Among them, the maximum and minimum normalization formula is used to normalize all data points, so as to normalize each data point in all data points to the interval of (0, 1); It is worth emphasizing that the other data points are complete data points, that is, there is no missing data at any data point; In this embodiment, the Euclidean distance formula is used to calculate the data point x to be filled. m With the complete data point x c distance; The Euclidean distance formula is specifically:

[0030] In the formula, X mi is the normalized ith data point in the data point to be filled, i=1-n, X ci is the normalized ith data among the other data points corresponding to the position of the data point to be filled, and n is the number of data among the data points to be filled; S2.3.3.2: Sort the distances to obtain k other data points with the smallest distances; In this step, through the above steps, the Euclidean distance value between the data point to be filled and each other data point can be obtained; the k data points most relevant to the data point to be filled can be determined through the Euclidean distance value; In the prior art, when determining the specific value of k, it is generally set based on experience, for example, k is set to any natural number between 5 and 8. However, the value of k set randomly affects the number of the most relevant data points involved in the subsequent filling. For different types of data points to be filled, the value of k set randomly affects the accuracy of data filling of missing values. Therefore, in view of the above technical problems and based on the characteristics of laboratory capability verification data collection, this embodiment proposes a method for determining a specific value of k to improve the data filling accuracy of the data points to be filled; Specifically, the specific value determination formula of k is:

[0031] In the formula, is the average of the five minimum distances, a is the number of missing values ​​in the data points to be filled, and a n is the number of data in each data point, b is the number of data points, r is the coefficient, in this embodiment, r is 100; By using the above formula to determine the specific value of k, the specificity of filling missing data in data points in laboratory proficiency testing is improved, and missing data in various types of data points can be filled in a targeted manner, thereby improving the accuracy of data filling for the data points to be filled.

[0032] S2.3.3.3: Fill the missing values ​​in the data points to be filled according to the k other data points with the smallest distances; In this step, the average of the k values ​​of each missing position among the other k data points with the smallest distance is used as the filling value to perform the missing value filling operation on the corresponding position.

[0033] In this step, during the data filling operation, the data type to be filled is first classified and judged, and then different data filling methods are selected according to different types of data, thereby improving the accuracy of data filling. At the same time, in the case where there are more data points to be filled, the operation of finding the data point with the highest correlation as the benchmark for data filling can effectively improve the accuracy of data filling.

[0034] S3: Conduct laboratory proficiency testing evaluation based on the data preprocessed by S2.

[0035] In this step, classical statistical methods are used to evaluate laboratory proficiency testing.

[0036] Embodiment 2, the present invention also provides an abnormal data processing system for laboratory proficiency verification, the system adopts the above-mentioned abnormal data processing method for laboratory proficiency verification, and the system comprises: The information acquisition module is used to obtain the information of laboratories participating in the proficiency testing.

[0037] A data preprocessing module, used for performing preprocessing operations on the laboratory test data in the laboratory information; The capability verification module is used to perform laboratory capability verification evaluation based on the data preprocessed by the data preprocessing module.

[0038] Embodiment 3, the present invention also provides an electronic device, including one or more processors and a memory.

[0039] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0040] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may run the program instructions to implement an abnormal data processing method for laboratory proficiency verification and / or other desired functions of any embodiment of the present application described above. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0041] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). The input device may include, for example, a keyboard, a mouse, etc. The output device may output various information to the outside, including early warning information, braking force, etc. The output device may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0042] Of course, for the sake of simplicity, components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0043] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to implement the functions of the abnormal data processing method for laboratory proficiency verification provided by any embodiment of the present application.

[0044] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0045] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor implements an abnormal data processing method for laboratory proficiency verification provided by any embodiment of the present application.

[0046] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0047] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0048] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing abnormal data for laboratory proficiency verification, characterized in that: The following steps are involved: S1: Obtain information of laboratories participating in proficiency testing; S2: performing a preprocessing operation on the laboratory test data in the laboratory information; The pre-processing operation includes: S2.1: Detect abnormal data points on the laboratory test data; S2.2: Delete the detected abnormal data points; S2.3: Fill in missing values ​​in the test data; The S2.3 is specifically: S2.3.1: Classify and judge the missing values. If the number of missing values ​​in the data point is greater than a preset value, proceed to S2.3.2; otherwise, proceed to S2.3.3; S2.3.2: Use the adjacent value averaging method to perform data filling operation on the missing values; S2.3.3: Use the relevant data filling method to fill in the missing values; S3: Conduct laboratory proficiency testing evaluation based on the data preprocessed by S2.

2. The method for processing abnormal data for laboratory proficiency testing according to claim 1, characterized in that: In S2.3.2, the formula for the data filling operation of the missing value by the adjacent value averaging method is: In the formula, x i is a missing value, x i-1 and x i+1 is the value of the missing value.

3. The abnormal data processing method for laboratory proficiency verification according to claim 2, characterized in that: The operation of filling the missing values ​​with the relevant data filling method is specifically as follows: S2.3.3.1: Normalize all data points, and then calculate the distance between the data point to be filled and other data points; S2.3.3.2: Sort the distances to obtain k other data points with the smallest distances; The specific value determination formula of k is: In the formula, is the average of the five minimum distances, a is the number of missing values ​​in the data points to be filled, and a n is the number of data in each data point, b is the number of data points, and r is the coefficient; S2.3.3.3: Fill in the missing values ​​in the data points to be filled based on the k other data points with the smallest distances.

4. The method for processing abnormal data for laboratory proficiency verification according to claim 3, characterized in that: The Euclidean distance formula is used to calculate the data point x to be filled m With the complete data point x c distance; The Euclidean distance formula is specifically: Where, X mi is the normalized ith data point in the data point to be filled, i=1-n, X ci is the normalized ith data among the other data points corresponding to the position of the data point to be filled, and n is the number of data in the data point to be filled.

5. The method for processing abnormal data for laboratory proficiency testing according to claim 1, characterized in that: In S2.3.1, the preset value is 1.

6. The method for processing abnormal data for laboratory proficiency testing according to claim 1, characterized in that: In S2.1, abnormal data points are detected based on the k-means clustering algorithm.

7. The method for processing abnormal data for laboratory proficiency testing according to claim 6, characterized in that: The S2.1 is specifically: S2.1.1: Calculate for each data point x in the laboratory test data i With the initial cluster center The distance between i ; S2.1.2: Determine the minimum distance between the test data point and the cluster center , according to the Assigning data points in the test data to corresponding clusters; S2.1.3: updating the cluster center; S2.1.4: Repeat S2.1.1-S2.1.3 until the cluster center and the objective function value remain unchanged; S2.1.5: Detect abnormal data points based on clustering results.

8. The method for processing abnormal data for laboratory proficiency testing according to claim 1, characterized in that: In S1, the laboratory information includes: laboratory quantity and laboratory test data.

9. The method for processing abnormal data for laboratory proficiency testing according to claim 1, characterized in that: In S2.1.1, each data point and the initial cluster center The distance between is calculated as: In the formula, x ij is the j-th test result of the i-th data point in the test data, is the jth dimension of the lth cluster center, and m is the number of data in each data point in the test data.

10. An abnormal data processing system for laboratory proficiency verification, characterized in that: The system adopts the abnormal data processing method for laboratory capability verification according to any one of claims 1 to 9, and the system comprises: The information acquisition module is used to obtain the information of laboratories participating in the proficiency testing. A data preprocessing module, used for performing preprocessing operations on the laboratory test data in the laboratory information; The capability verification module is used to perform laboratory capability verification evaluation based on the data preprocessed by the data preprocessing module.

Images