Cross-laboratory heterogeneous data exchange and intelligent cleaning method

Abstract

The invention provides a cross-laboratory heterogeneous data exchange and intelligent cleaning method, and relates to the technical field of data cleaning, and the method comprises the following steps: carrying out the data collection from a plurality of laboratory heterogeneous systems according to the data type, and transmitting the data to a data exchange platform through a network; after the data exchange platform obtains the data, data cleaning is carried out on the data, and whether the data are repeated or not is judged; if so, generating a unique identifier to remove duplication, and filling missing items in the data according to the data type; performing standardization processing on the filled data, and storing the data according to metadata classification; and distributing the stored data to a plurality of laboratories according to a preset data planning rule. According to the method, standardization and automation of the whole process of data acquisition, cleaning, storage and distribution are achieved, different scene requirements of different laboratories are met, the manual operation cost is remarkably reduced, and reliable support is provided for efficient exchange and collaborative application of cross-laboratory data.

Description

Technical Field

[0001] This invention relates to the technical field of data cleaning, and more particularly to a method for cross-laboratory heterogeneous data exchange and intelligent cleaning. Background Technology

[0002] With the advancement of laboratory informatization, different laboratories often deploy heterogeneous data systems (such as sample management systems, experimental analysis systems, and testing record systems) according to their own business needs. These systems have significant differences in data formats, storage specifications, and field definitions, which leads to many bottlenecks in cross-laboratory data exchange.

[0003] In existing technologies, cross-laboratory data processing mainly suffers from the following problems: The data types (numerical, categorical, time, etc.), field naming rules, and data formats generated by the heterogeneous systems in various laboratories are inconsistent. Direct transmission of these data can easily lead to problems such as data parsing failure and field mismatch, making efficient data exchange impossible.

[0004] Traditional methods for identifying duplicate data often rely on single-field comparisons, failing to consider scenarios where there are "no globally unique fields" in laboratory data. Furthermore, direct field comparisons are inefficient and suffer from severe response delays in massive data scenarios. Alternatively, the collected raw data often contains duplicate data (such as multiple entries of the same experimental sample), missing data (such as missing experimental parameters), and abnormal formats. Existing cleaning methods mostly involve manual screening or single-rule processing, which are inefficient and prone to errors, making it difficult to meet the cleaning needs of large-scale laboratory data.

[0005] Therefore, there is an urgent need for a technical solution that can adapt to the characteristics of heterogeneous data across laboratories, balance the accuracy and efficiency of data cleaning, and achieve standardized data flow throughout the entire process, in order to solve the above-mentioned defects in existing technologies. Summary of the Invention

[0006] In view of the technical problems mentioned in the background section above, a method for cross-laboratory heterogeneous data exchange and intelligent cleaning is provided.

[0007] The technical means employed in this invention are as follows: A method for cross-laboratory heterogeneous data exchange and intelligent cleaning includes the following steps: Step 1: Collect data from multiple heterogeneous laboratory systems according to data type and send it to the data exchange platform via network; Step 2: After acquiring the data, the data exchange platform cleans the data and determines whether the data is duplicated. Step 3: If duplicates are found, generate a unique identifier to remove duplicates and fill in the missing items in the data according to the data type; if the data is numerical, fill it with the mean or median; if the data is categorical, fill it with the category that appears most frequently. Step 4: Standardize the filled data and store it according to metadata classification.

[0008] Furthermore, step 3 includes the following steps: Step 31: Determine the criteria for determining duplicates; the criteria for determining duplicates include: any one of a single unique field or a combination of multiple fields; Step 32: Traverse the data and perform hash calculations on the duplicate determination criteria for each record; Step 33: If the criteria for determining duplicate records are completely identical for two or more records, then the records are determined to be duplicates.

[0009] Furthermore, the hash calculation formula is as follows: ; in, This indicates the hash value used to determine if a record is duplicated. Represents the hash algorithm function; This represents a string concatenation function, used to combine multiple fields sequentially into a single complete string; Indicates a single, unique field; This indicates a combination of multiple fields.

[0010] Furthermore, the data acquisition from multiple heterogeneous laboratory systems in step 1, based on the data type, includes one or more combinations of timed acquisition or real-time acquisition; if timed acquisition is used, the acquisition period is from 1 minute to 24 hours.

[0011] Furthermore, in step 4, Z-score normalization or Min-Max normalization is used.

[0012] Furthermore, in step 5, the stored data is distributed at a frequency of 1 minute to 24 hours.

[0013] Compared with the prior art, the present invention has the following advantages: The method of this invention achieves standardization and automation of the entire process of data acquisition, cleaning, storage and distribution. It not only adapts to the different scenario needs of different laboratories, but also significantly reduces the cost of manual operation, providing reliable support for efficient data exchange and collaborative application across laboratories. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] like Figure 1 As shown, this invention provides a method for cross-laboratory heterogeneous data exchange and intelligent cleaning, comprising the following steps: First, data acquisition is performed. For heterogeneous systems deployed across multiple laboratories, such as biological testing laboratories and chemical analysis laboratories (e.g., sample management systems, experimental data recording systems, and testing instrument terminals), data type classification standards are preset (including numerical data such as experimental concentration values, categorical data such as sample categories, and time-based data such as testing time). Then, the acquisition method is selected based on the timeliness requirements of each laboratory. For core experimental data with high real-time requirements (e.g., online testing data), a real-time acquisition mode is used, capturing raw data generated by heterogeneous systems in real time via API interface. For non-core statistical data (e.g., daily experimental summary data), a timed acquisition mode is used, with a preset acquisition cycle of 1 hour (configurable within a range of 1 minute to 24 hours), periodically extracting data through a scheduled task tool. All acquired raw data is encrypted and transmitted over the network to a preset data exchange platform, with TCP / IP protocol used to ensure data integrity during transmission.

[0019] After receiving the raw data, the data exchange platform immediately initiates the data cleaning process, prioritizing data duplication detection. During duplication detection, step 31 is executed first to determine the criteria for duplication determination: if a globally unique field exists in the laboratory data (such as the sample number "Sample-20240501-001"), this field is directly used as the single unique field (F1) as the criterion; if no globally unique field exists, a combination of multiple fields, "sample name + test item + test time," is selected as the criterion (i.e., F1 = sample name, F2 = test item, F3 = test time, n = 3). Then, step 32 is executed for hash calculation. First, the fields of the criterion are concatenated sequentially using the Concat string concatenation function, and then the concatenated string is input into the MD5 hash algorithm function to calculate a 32-bit hexadecimal hash value (such as "e89f495f0096d02729e379580e871926"). Finally, step 33 is executed, which iterates through the hash values ​​of all the original data. If two or more records have completely identical hash values, they are determined to be duplicate data. At this time, a unique identifier (such as "Unique-20240501-0001") is generated to remove duplicate data, and only one complete record is retained.

[0020] In this application, the hash calculation formula is: ; in, This indicates the hash value used to determine if a record is duplicated. Represents the hash algorithm function; This represents a string concatenation function, used to combine multiple fields sequentially into a single complete string; Indicates a single, unique field; This indicates a combination of multiple fields.

[0021] After deduplication, missing data is imputed: if the missing data is numerical (e.g., experimental absorbance values), the median of valid data for that field is 0.85, and 0.85 is used to impute the missing data; if the missing data is categorical (e.g., sample source), the most frequent category for that field is "equipment name," and "equipment name" is used to impute the missing data. If no high-frequency category exists, the preset "unknown" label is used. If the missing rate of a field exceeds 30%, that field is deleted to avoid affecting the overall data quality.

[0022] After missing data are filled, the data undergoes standardization: if the data represents experimental error values ​​requiring cross-laboratory comparison, the Z-score standardization algorithm is used, using the formula: ; in, Represents the original data; This represents the mean; Indicates the standard deviation; transforms the data into a standard normal distribution; if the data are concentration values ​​that need to be limited to a specific range (e.g., 0- If the value is 0, then the Min-Max normalization algorithm is used, using the formula: ; Map the data to After standardization, the data is stored in the corresponding partition node of the distributed database according to the preset metadata classification rules (such as classification by "sample type-detection item-data source"). The storage format adopts UTF-8 encoding, and date data is uniformly in the format of "YYYY-MM-DDHH:MM:SS".

[0023] Finally, the data allocation operation is performed, with a preset allocation frequency of 2 hours (within the range of 1 minute to 24 hours). The data exchange platform allocates the stored standardized data to the application systems of various laboratories (such as the data analysis platform of Laboratory A and the sample traceability system of Laboratory B) according to this frequency, using an access control mechanism. During the allocation process, the access information of the receiving end is verified, and data is only transmitted to authorized nodes to ensure data security. The entire process is automated and requires no manual intervention. An alarm is triggered only when the proportion of abnormal data exceeds 5%, at which point staff will verify the data source and handle the situation.

[0024] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0025] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0026] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0027] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0028] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0029] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0030] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for cross-laboratory heterogeneous data exchange and intelligent cleaning, characterized in that, Includes the following steps: Step 1: Collect data from multiple heterogeneous laboratory systems according to data type and send it to the data exchange platform via network; Step 2: After the data exchange platform acquires the data, it performs data cleaning and determines whether the data is duplicated. Step 3: If duplicates are found, generate a unique identifier to remove duplicates and fill in the missing items in the data according to the data type; if the data is numerical, fill it with the mean or median; if the data is categorical, fill it with the category that appears most frequently. Step 4: Standardize the filled data and store it according to metadata classification. Step 5: Distribute the stored data to multiple laboratories according to the preset data planning rules.

2. The method for cross-laboratory heterogeneous data exchange and intelligent cleaning according to claim 1, characterized in that, Step 3 includes the following steps: Step 31: Determine the criteria for determining duplicates; the criteria for determining duplicates include: any one of a single unique field or a combination of multiple fields; Step 32: Traverse the data and perform hash calculations on the duplicate determination criteria for each record; Step 33: If the criteria for determining duplicate records are completely identical for two or more records, then the records are determined to be duplicates.

3. The method for cross-laboratory heterogeneous data exchange and intelligent cleaning according to claim 2, characterized in that, The hash calculation formula is as follows: ； in, This indicates the hash value used to determine if a record is duplicated. Represents the hash algorithm function; This represents a string concatenation function, used to combine multiple fields sequentially into a single complete string; Indicates a single, unique field; This indicates a combination of multiple fields.

4. The method for cross-laboratory heterogeneous data exchange and intelligent cleaning according to claim 1, characterized in that, Step 1 involves data acquisition from multiple heterogeneous laboratory systems based on data type, including one or more combinations of timed acquisition or real-time acquisition; if timed acquisition is used, the acquisition period is from 1 minute to 24 hours.

5. The method for cross-laboratory heterogeneous data exchange and intelligent cleaning according to claim 1, characterized in that, In step 4, Z-score normalization or Min-Max normalization is used.

6. The method for cross-laboratory heterogeneous data exchange and intelligent cleaning according to claim 1, characterized in that, In step 5, the stored data is distributed at a frequency of 1 minute to 24 hours.

Images