A peakfit data automatic processing method and system

Abstract

This invention provides a method and system for automated PeakFit data processing, relating to the field of data processing technology. The method includes: collecting result files in the PeakFit output directory using regular expression matching; verifying the format and completeness of the result files; extracting key parameters from the verified result files and generating a parameter association table between the key parameters and metadata; cleaning the key parameters in the parameter association table using statistical features and dynamic thresholds, and standardizing the format of the cleaned key parameters; verifying the accuracy of the key data after data cleaning and format standardization using cross-validation; and organizing the verified key parameters into an Excel worksheet according to a preset format.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a PeakFit data automation processing method and system. Background Technology

[0002] With the increasing demands for data processing efficiency and accuracy in scientific research and engineering applications, peak fitting technology has been widely used in fields such as spectral analysis, chromatographic analysis, thermal analysis, and biomedical analysis. Peak fitting not only helps researchers quickly identify characteristic peaks in data, but also provides support for subsequent structural analysis, chemical composition analysis, and pharmacodynamic studies through parameter extraction. Among numerous peak fitting software programs, PeakFit is widely used due to its powerful functions, good compatibility, and wide applicability.

[0003] Currently, PeakFit software provides basic functions for data processing and result output. After completing peak fitting, users can save a single result file or copy the fitting parameters to Excel or other data processing software. For small amounts of data, this method can meet basic needs.

[0004] However, existing technologies mainly rely on manual copying and pasting or saving files individually, which is inefficient and prone to errors. Especially when processing large amounts of data in batches, it not only consumes a lot of time and effort, but also easily leads to omissions and formatting problems. At the same time, the lack of automation and standardization makes it difficult to guarantee the accuracy and consistency of data, which seriously affects data processing efficiency. Summary of the Invention

[0005] To address the technical problems of existing technologies that rely primarily on manual copying and pasting or saving files individually, which are inefficient and prone to errors, especially when processing large amounts of data in batches, this invention provides a PeakFit automated data processing method and system.

[0006] The technical solutions provided by the embodiments of the present invention are as follows:

[0007] First aspect:

[0008] An embodiment of the present invention provides an automated data processing method for PeakFit, comprising:

[0009] S1: Collect the result files in the PeakFit output directory using regular expression matching;

[0010] S2: Verify the format and integrity of the result file;

[0011] S3: Extract the key parameters from the verified result file and generate a parameter association table between the key parameters and the metadata;

[0012] S4: Clean the key parameters in the parameter association table using statistical features and dynamic thresholds, and standardize the format of the cleaned key parameters.

[0013] S5: Verify the accuracy of key parameters after data cleaning and format standardization through cross-comparison verification;

[0014] S6: Organize the key parameters after accuracy verification into an Excel worksheet according to the preset format.

[0015] The second aspect:

[0016] An embodiment of the present invention provides a PeakFit data automation processing system, comprising:

[0017] processor;

[0018] A memory storing computer-readable instructions that, when executed by the processor, implement the PeakFit data automation processing method as described in the first aspect.

[0019] Third aspect:

[0020] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the PeakFit data automation processing method as described in the first aspect.

[0021] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0022] In this invention, PeakFit output files are automatically collected and verified, key parameters are extracted and a parameter correlation table is generated, and the processing results are directly imported into Excel in a preset format. This effectively avoids the inefficiency and error-proneness caused by manual copying and pasting or saving files one by one. Simultaneously, data cleaning and format standardization are combined, and a cross-validation mechanism ensures the consistency and accuracy of key parameters. This provides automation and standardization support while achieving efficient batch data processing, significantly improving the reliability and overall efficiency of data processing. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a PeakFit data automation processing method provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of a PeakFit automated data processing system provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0027] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0028] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0029] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0030] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0031] Reference manual attached Figure 1 The diagram shows a flowchart of a PeakFit data automation processing method provided by an embodiment of the present invention.

[0032] This invention provides a method for automated PeakFit data processing, which can be implemented by a PeakFit data automation processing device, which can be a terminal or a server. The processing flow of the PeakFit data automation processing method may include the following steps:

[0033] S1: Collect the result files in the PeakFit output directory using regular expression matching.

[0034] Regular expression matching (GEM) is a technique that uses rules to check whether text (such as filenames, logs, or data lines) conforms to a expected format and can extract key information from it.

[0035] In one possible implementation, S1 specifically includes:

[0036] S101: Create a named regular expression suitable for the result file.

[0037] For example, the named regular expression is: PeakFit20250828_Batch1_SAMPLE-01.csv. Prefix=PeakFit, Date=20250828, Batch=Batch1, Sample=SAMPLE-01, Extension=csv.

[0038] S102: Use named regular expressions to perform regular expression matching on each result file in the PeakFit output directory.

[0039] S103: Determine if each result file successfully matches the regular expression. If yes, extract the metadata of the result file. Otherwise, add the result file to the quarantine area.

[0040] S104: Store the metadata in a temporary SQLite database to form a target mapping table that represents the mapping relationship between the result file and the metadata. The metadata includes the file path, file name, file creation time, experiment date, and batch ID.

[0041] Specifically, the extracted metadata and file paths are written into a temporary SQLite database to form a "file-to-metadata mapping table". This allows for quick retrieval of data from a particular sample or a day's experiment.

[0042] It should be noted that the experiment date is obtained by parsing the field in the regular expression rule, and the batch ID is obtained by parsing the identifier part in the regular expression rule.

[0043] In this embodiment of the invention, regular expression matching can filter out files with non-compliant or incorrect names, placing them into an "isolation zone" to ensure that subsequent processing inputs are high-quality, compliant data. Simultaneously, a single regular expression match can extract key metadata such as experiment date, batch ID, and sample ID, eliminating the need for manual input or additional scripts to split strings and reducing human error.

[0044] S2: Verify the format and integrity of the result file.

[0045] In one possible implementation, the verification specifically includes: structural entropy verification for format verification and semantic topology verification for integrity verification.

[0046] Among them, structural entropy verification is a file format integrity detection method. It borrows the idea of ​​information entropy to measure the "disorder" of the file structure. Here, structural entropy is based on the distribution of different delimiters (such as commas, tabs, spaces, and semicolons) in the file to quantify the consistency of the file format.

[0047] Semantic topology validation is a method for checking the format and integrity of files at the content level. It focuses not on delimiters (that's the task of structural entropy), but on whether the column names and field meanings in the file are consistent with the expected domain semantics.

[0048] In one possible implementation, S2 specifically includes:

[0049] S201: By using segmented sampling, the first N lines, middle N lines, and last N lines of each result file are summarized, and the distribution of delimiters in all the summarized lines is statistically analyzed.

[0050] Specifically, the delimiters include commas, tabs, spaces, and semicolons.

[0051] It should be noted that if the number of rows is not full, the entire data should be processed directly.

[0052] In this embodiment of the invention, the overall characteristics of the file format can be quickly obtained without reading the entire large file, which improves efficiency and avoids missing abnormal delimiters that may exist at the beginning, end or in the middle, thereby ensuring the representativeness of the sample.

[0053] S202: Based on the delimiter distribution summary results, calculate the structural entropy value of each result file:

[0054]

[0055] Where h represents the structural entropy value of the result file, Let represent the probability of the i-th delimiter type appearing in the file, n represent the total number of delimiter types, and log represents the logarithm function to the base 2.

[0056] S203: Query the mean and standard deviation of structural entropy in historical valid result files, and calculate the dynamic threshold of structural entropy based on the mean and standard deviation of structural entropy.

[0057]

[0058] in, Represents the dynamic threshold of structural entropy. This represents the average structural entropy of historical valid result files. The standard deviation of the structural entropy of historical valid result files.

[0059] It should be noted that if there is no historical data, the default threshold (such as a preset entropy value based on the distribution of common delimiters) will be used.

[0060] In this embodiment of the invention, the judgment criteria can be adaptively adjusted according to the distribution of historical valid files, rather than relying on a fixed threshold, thereby improving the system's adaptability to different experimental environments and data habits.

[0061] S204: Determine whether the structural entropy value of each result file is less than the dynamic threshold of structural entropy. If so, mark the result file as a result file to be verified and proceed to S205. Otherwise, mark the result file as structurally damaged and place it in the isolation area for manual processing.

[0062] It should be noted that by automatically identifying and isolating obviously damaged files in advance, the spread of errors in subsequent steps is reduced, thus ensuring the quality of data processed later.

[0063] S205: A pre-built standardized keyword vector library for the PeakFit domain.

[0064] Specifically, we collected column names from PeakFit's official documentation and commonly used industry column names (such as `Peak Area`, `Peak Center`, `Peak Height`, and `FWHM`) and trained them into a standardized vector library using the Word2Vec model (window=2, vector_size=100).

[0065] S206: Use pre-trained Word2Vec word embeddings to vectorize the column names in the result file to be verified, and obtain column name vectors.

[0066] S207: Calculate the cosine similarity matrix between the column name vector and the keyword vectors in the standardized keyword vector library.

[0067] S208: Based on the cosine similarity matrix, determine whether the difference between the highest and second-highest similarity scores corresponding to each column name vector is less than a preset difference. If so, place the column name vectors in an isolation area for manual processing. Otherwise, determine that the match is correct, map the column name vectors to the standardized keyword vector library, and complete the format and completeness verification.

[0068] Optionally, the default difference is 0.1.

[0069] It should be noted that those skilled in the art can set the size of the preset difference according to actual needs, and this invention does not limit this.

[0070] In this embodiment of the invention, a "difference threshold" mechanism is introduced to avoid erroneous matching caused by semantic ambiguity, ensuring that only high-confidence matches are accepted, while ambiguous column names are isolated and manually processed, thereby improving system reliability.

[0071] S3: Extract key parameters from the verified result file and generate a parameter association table between the key parameters and metadata.

[0072] It should be noted that the metadata specifically includes file path, file name, experiment date, and batch ID.

[0073] In one possible implementation, the key parameters specifically include: peak area, peak center value, and peak height.

[0074] S3 specifically includes:

[0075] S301: Based on the mapping result of the column name vector, locate the key parameter columns in the result file after the verification is completed.

[0076] It's important to note that the standardized column name mapping automatically or semi-automatically maps these diverse original column names to a standardized set of keywords (i.e., domain-unified standard column names). This ensures that all subsequent steps are processed under consistent semantics, unaffected by differences in document syntax.

[0077] S302: Use Python's pandas library to read the numerical data of key parameter columns and store the numerical data as a DataFrame data structure.

[0078] Python, a high-level programming language, was first released by Guido van Rossum in 1991. It is known for its simplicity, readability, and high development efficiency, and is one of the most widely used languages ​​today.

[0079] Pandas is an open-source data analysis and processing library for Python. It is one of the most commonly used data analysis tools, especially suitable for processing structured data (tables, time series, etc.). It has wide applications in data science, finance, scientific research, and automated reporting.

[0080] DataFrame is one of the core data structures in Python's Pandas library. It can be understood as a labeled two-dimensional table, similar to an Excel spreadsheet, an SQL table, or a data frame in R.

[0081] S303: Associate the numerical data of the DataFrame data structure with the target mapping table using the sample ID as the key to generate a parameter association table of key parameters and metadata.

[0082] It's important to note that in PeakFit's automated processing workflow, the key parameter and metadata association table is a crucial data structure that bridges the gap between different parts of the data. Its function is to link the experimental results (key parameters) within the file content with the file's descriptive information (metadata), forming a complete view of the experimental data.

[0083] In this embodiment of the invention, the association operation integrates experimental results and file metadata, which not only improves the integrity and traceability of the data, but also provides a unified data foundation for subsequent statistical analysis, cleaning and verification, and automated report generation. It is a key link in the entire automated processing chain.

[0084] S4: Clean the key parameters in the parameter association table using statistical features and dynamic thresholds, and then standardize the format of the cleaned key parameters.

[0085] In one possible implementation, S4 specifically includes:

[0086] S401: Calculate the mean, standard deviation, first quartile, and third quartile of each parameter column corresponding to the key parameter in the parameter association table.

[0087] In this embodiment of the invention, by calculating these statistical characteristics, a comprehensive understanding of the overall distribution of the parameter column can be obtained. This provides a basis for subsequent determination of data distribution type and setting cleaning thresholds, avoiding reliance on experience or subjective human judgment and improving the scientific rigor of the cleaning process.

[0088] S402: Use the Shapiro-Wilk test to determine the distribution type of each parameter column, where the distribution type includes normal distribution and non-normal distribution.

[0089] The Shapiro-Wilk test is a statistical method used to test whether a set of data follows a normal distribution.

[0090] It should be noted that if the parameter column has few data points (e.g., less than 20 rows), it is considered to be non-normally distributed by default.

[0091] S403: For normally distributed parameter columns, use the Three Sigma principle for data cleaning. For non-normally distributed parameter columns, use the box plot criterion for data cleaning.

[0092] Specifically, data cleaning using the Three Sigma principle involves placing data outside the range μ±3σ in an isolated region. Data cleaning using the box plot criterion involves removing data falling outside the range [Q1−1.5IQR, Q3+1.5IQR] and placing it in an isolated region.

[0093] In this embodiment of the invention, different distribution types require different outlier detection methods. By testing whether the data follows a normal distribution, the "three sigma principle" or the "box plot criterion" can be selected to make the cleaning method more consistent with the actual distribution characteristics of the data, thereby reducing false positives and false negatives. For cases with few data points, non-normality is directly assumed to avoid the instability of small sample statistical tests.

[0094] S404: Standardize the various parameter columns after cleaning. The standardization process includes converting parameter values ​​from different sources into a unified standard unit and converting all parameter values ​​into floating-point numbers.

[0095] For example, the peak area is uniformly measured in cm², the peak center is uniformly measured in nm, and the peak height is uniformly measured in au.

[0096] S5: Verify the accuracy of key parameters after data cleaning and format standardization through cross-comparison verification.

[0097] Cross-validation is a result verification method used to ensure the accuracy and reliability of data after cleaning and standardization.

[0098] In one possible implementation, S5 specifically includes:

[0099] S501: Based on the sample ID in the PeakFit result file, determine whether the key parameters after data cleaning and format standardization are duplicate sample data. If yes, proceed to S502. Otherwise, proceed to S503.

[0100] It should be noted that samples with the same ID but different batch IDs are also considered duplicate samples.

[0101] In this embodiment of the invention, by identifying whether the same SampleID (even if from different batches) is repeated, samples can be divided into two categories: "with repetition" and "without repetition," thereby selecting the most suitable verification method. This avoids a one-size-fits-all approach and ensures that the verification method is more closely aligned with the characteristics of the data.

[0102] S502: Calculate the relative error between a pair of duplicate sample data and determine whether the relative error is less than a preset relative error. If so, determine that the pair of duplicate sample data is valid data. Otherwise, mark the pair of duplicate sample data as data to be checked and return to S4 for re-cleaning.

[0103] Optionally, the specific method for calculating the relative error between two repeated samples is as follows:

[0104]

[0105] in, This represents the relative error between two repeated samples. This represents the key parameter measurement value of the first replicated sample. This represents the key parameter measurement value of the corresponding second replicate sample. This represents absolute value operations.

[0106] S503: For data with no duplicate samples, use a pre-trained association model to calculate the predicted value, determine the absolute error between the predicted and actual values, and check if the absolute error is less than the preset absolute error. If so, the data with no duplicate samples is considered valid. Otherwise, mark the data with no duplicate samples as data to be checked and return to S4 for re-cleaning.

[0107] It should be noted that those skilled in the art can set the magnitude of the preset relative error and preset absolute error according to actual needs, and this invention does not limit this.

[0108] For example, the formula for predicting peak area is: ,in, H represents the predicted peak area, W represents the peak height, and k represents a constant obtained by fitting historical experimental data. The k value is obtained by linear regression fitting based on historical data. If no historical data is available, the default k value (industry standard) is used. The absolute error in calculating the peak area is: ,in, This represents the absolute error of the peak area. Indicates the predicted peak area. This represents the actual measured peak area. This indicates absolute value calculation. If the absolute error is less than 0.1, the single sample data is considered valid. Otherwise, it is marked as data to be verified.

[0109] In this embodiment of the invention, a small error between the predicted value and the actual value indicates that the data is reasonable, while a large error triggers a review. This allows for verification even with a single sample, improving the system's coverage and robustness.

[0110] S6: Organize the key parameters after accuracy verification into an Excel worksheet according to the preset format.

[0111] In one possible implementation, S6 specifically includes:

[0112] S601: Set the output template for the Excel worksheet, which includes the naming rules and column order of the Excel worksheet.

[0113] Specifically, the naming convention is PeakFit_Results_YYYYMMDD_BatchID, which is an automated naming template. YYYYMMDD is replaced with the current experiment date, and BatchID is replaced with the batch number, ensuring that each results report corresponds one-to-one with the corresponding experiment date and batch. The columns are in the following order: Sample ID, Peak Area, Peak Center, and Peak Height.

[0114] In this embodiment of the invention, a preset naming rule ensures that each result file clearly corresponds to a specific experimental date and batch, preventing naming confusion or file loss. This automated naming also facilitates later archiving, searching, and traceability.

[0115] S602: Automatically fill the key parameters after accuracy verification into the Excel worksheet according to the output template.

[0116] S603: Based on the completed Excel worksheet, generate an Excel result file and store the Excel result file in the specified directory.

[0117] In this embodiment of the invention, key parameters are automatically populated into the template, avoiding the manual copying, pasting, and sorting process, significantly reducing errors caused by human operation and ensuring the accuracy and integrity of the data. Simultaneously, the generated Excel files are uniformly stored in a designated directory, which can then be archived and distributed as the final "experimental results." Whenever it is necessary to check a particular batch of data, it can be quickly located through naming rules and storage directories, improving work efficiency.

[0118] In one possible implementation, the process after S6 includes:

[0119] S7: Calculate the hash value of the Excel result file and verify the integrity of the file when calling or transmitting the Excel result file based on the hash value.

[0120] Optionally, the hash algorithm chosen is SHA-256. SHA-256 is existing technology and will not be described in detail here.

[0121] It's important to note that each Excel file generates a unique SHA-256 hash value. If even a single byte of the file content is tampered with or corrupted, the hash value will change. This allows for quick determination of file modification status by comparing the hash values ​​after accessing or transferring the file, ensuring data integrity.

[0122] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0123] In this invention, PeakFit output files are automatically collected and verified, key parameters are extracted and a parameter correlation table is generated, and the processing results are directly imported into Excel in a preset format. This effectively avoids the inefficiency and error-proneness caused by manual copying and pasting or saving files one by one. Simultaneously, data cleaning and format standardization are combined, and a cross-validation mechanism ensures the consistency and accuracy of key parameters. This provides automation and standardization support while achieving efficient batch data processing, significantly improving the reliability and overall efficiency of data processing.

[0124] Reference manual attached Figure 2 The diagram shows a structural schematic of a PeakFit data automation processing system provided by the present invention.

[0125] The present invention also provides a PeakFit data automation processing system 20, applied to the above-mentioned PeakFit data automation processing method, comprising:

[0126] Processor 201.

[0127] The memory 202 stores computer-readable instructions, which, when executed by the processor 201, implement the PeakFit data automation processing method as described in the method embodiment.

[0128] The PeakFit data automation processing system 20 provided by this invention can execute the PeakFit data automation processing method described above and achieve the same or similar technical effects. To avoid repetition, this invention will not elaborate further.

[0129] It should be understood that the processor in the embodiments of the present invention can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0130] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0131] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0132] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0133] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0134] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0137] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0140] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] This invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the PeakFit data automation processing method as described in the method embodiment.

[0142] The present invention provides a computer-readable storage medium that can implement the steps and effects of the PeakFit data automation processing method described in the above method embodiments. To avoid repetition, the present invention will not repeat them.

[0143] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0144] The following points need to be explained:

[0145] (1) The accompanying drawings of the embodiments of the present invention only involve the structures involved in the embodiments of the present invention. Other structures can refer to the general design.

[0146] (2) For clarity, the thickness of layers or regions is enlarged or reduced in the drawings used to describe embodiments of the invention, i.e., these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element or there may be intermediate elements.

[0147] (3) Where there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0148] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automated processing of PeakFit data, characterized in that, include: S1: Collect the result files in the PeakFit output directory using regular expression matching; S2: Verify the format and integrity of the result file; S3: Extract the key parameters from the verified result file and generate a parameter association table between the key parameters and the metadata; S4: Clean the key parameters in the parameter association table using statistical features and dynamic thresholds, and standardize the format of the cleaned key parameters. S5: Verify the accuracy of key parameters after data cleaning and format standardization through cross-comparison verification; S6: Organize the key parameters after accuracy verification into an Excel worksheet according to the preset format; Specifically, the verification includes: structural entropy verification for format verification and semantic topology verification for integrity verification; Specifically, S2 includes: S201: By using segmented sampling, the first N lines, the middle N lines, and the last N lines of each of the result files are summarized, and the distribution of delimiters in all the summarized lines is statistically analyzed. S202: Based on the delimiter distribution summary results, calculate the structural entropy value of each of the result files; S203: Query the mean and standard deviation of the structural entropy of historical valid result files, and calculate the dynamic threshold of structural entropy based on the mean and standard deviation of the structural entropy; S204: Determine whether the structural entropy value of each of the result files is less than the structural entropy dynamic threshold; if so, mark the result file as a result file to be verified and proceed to S205; otherwise, mark the result file as structurally damaged and place it in the isolation area for manual processing. S205: A pre-built standardized keyword vector library for the PeakFit domain; S206: Using pre-trained Word2Vec word embeddings, vectorize the column names in the result file to be verified to obtain column name vectors; S207: Calculate the cosine similarity matrix between the column name vector and the keyword vectors in the standardized keyword vector library; S208: Based on the cosine similarity matrix, determine whether the difference between the highest similarity and the second highest similarity corresponding to each column name vector is less than a preset difference; if so, place the column name vector in the isolation area for manual processing; otherwise, determine that the match is correct, map the column name vector to the standardized keyword vector library, and complete the format and integrity verification.

2. The PeakFit data automated processing method according to claim 1, characterized in that, S1 specifically includes: S101: Create a named regular expression suitable for the result file; S102: Using the named regular expression, perform regular expression matching on each result file in the PeakFit output directory; S103: Determine whether each of the result files has successfully matched the regular expression; if so, extract the metadata of the result file; otherwise, add the result file to the isolation area. S104: Store the metadata in an SQLite temporary database to form a target mapping table representing the mapping relationship between the result file and the metadata, wherein the metadata includes file path, file name, file creation time, experiment date, and batch ID.

3. The PeakFit data automated processing method according to claim 1, characterized in that, The key parameters specifically include: peak area, peak center value, and peak height; S3 specifically includes: S301: Based on the mapping result of the column name vector, locate the key parameter columns in the result file after the verification is completed; S302: Use Python's pandas library to read the numerical data of the key parameter column and store the numerical data in a DataFrame data structure; S303: Associate the numerical data of the DataFrame data structure with the target mapping table using the sample ID as the key to generate a parameter association table between the key parameters and the metadata.

4. The PeakFit data automated processing method according to claim 1, characterized in that, S4 specifically includes: S401: Calculate the mean, standard deviation, first quartile, and third quartile of each parameter column corresponding to the key parameter in the parameter association table; S402: Use the Shapiro-Wilk test to determine the distribution type of each parameter column, wherein the distribution type includes normal distribution and non-normal distribution; S403: For the normally distributed parameter series, perform data cleaning operations using the three sigma principle; for the non-normally distributed parameter series, perform data cleaning operations using the box plot criterion. S404: Standardize the various parameter columns after cleaning, wherein the standardization process includes: converting parameter values ​​from different sources into a unified standard unit and converting all parameter values ​​into floating-point numbers.

5. The PeakFit data automated processing method according to claim 1, characterized in that, S5 specifically includes: S501: Based on the sample ID of the PeakFit result file, determine whether the key parameters after data cleaning and format standardization are duplicate sample data; if yes, proceed to S502; otherwise, proceed to S503. S502: Calculate the relative error between a pair of duplicate sample data and determine whether the relative error is less than a preset relative error; if so, determine that the pair of duplicate sample data is valid data; otherwise, mark the pair of duplicate sample data as data to be checked and return to S4 to clean again; S503: For non-repeating sample data, a pre-trained association model is used to calculate the predicted value, determine the absolute error between the predicted value and the actual value, and determine whether the absolute error is less than the preset absolute error; if so, the non-repeating sample data is determined to be valid; otherwise, the non-repeating sample data is marked as the data to be checked, and the process returns to S4 for re-cleaning.

6. The PeakFit data automated processing method according to claim 1, characterized in that, S6 specifically includes: S601: Set the output template of the Excel worksheet, wherein the output template includes the naming rules and column order of the Excel worksheet; S602: Automatically fill the key parameters after accuracy verification into the Excel worksheet according to the output template; S603: Based on the completed Excel worksheet, generate an Excel result file and store the Excel result file in the specified directory.

7. The PeakFit data automated processing method according to claim 6, characterized in that, Following S6, the following is also included: S7: Calculate the hash value of the Excel result file and verify the integrity of the file when calling or transmitting the Excel result file based on the hash value.

8. A PeakFit data automation processing system, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the PeakFit data automation processing method as described in any one of claims 1 to 7.

Images