Data processing system based on sample library

Abstract

The invention discloses a data processing system based on a sample library. The system comprises a data acquisition and import module; the data acquisition module is responsible for acquiring data from different port biological libraries and importing the data into the system; the data standardization and conversion module is responsible for uniformly converting the collected data of different ports into a system standard format; the data storage and management module is responsible for storing standardized data and providing data retrieval and management functions; the data interface and sharing module is responsible for providing a unified AP I interface and is used for data sharing among biological library samples of different ports; through the data acquisition and import module, original data can be acquired from multiple port biological libraries, and the data is cleaned, standardized and converted through the data standardization and conversion module, so that the problem of data format difference between different ports is solved. And the data integration and analysis module further merges the dispersed data into a unified database, so that the data can be seamlessly jointed and shared among different sample libraries.

Description

Technical Field

[0001] The present invention relates to the technical field of sample library data processing, and in particular to a system for sample library data processing. Background Art

[0002] A biobank is a warehouse that centrally stores human biological samples for analyzing disease cases and studying life sciences. Common samples include organs, tissues, blood, body fluids, or processed derivatives such as DNA, RNA, and proteins. These samples are extremely important for medical research. Through the sample bank, new treatment pathways can be explored, new diagnostic methods can be developed, and new drugs can be studied at the molecular level. The construction of a biobank has become the key to supporting modern translational medicine and scientific research. With the development of automated programs and computer technology, the storage form of biobanks has changed from small records decades ago to today's information system management. The information construction of the sample bank mainly includes the informationization of sample collection, storage, management, and analysis. Through information construction, comprehensive recording, tracking, and query of sample information can be achieved, improving the efficiency and accuracy of sample management.

[0003] In the management and research process of biobanks, a large amount of biological sample data is often stored in multiple different platform ports, such as hospitals, scientific research institutions, biobanks, etc. There are large differences in the data formats between these different ports, which makes it difficult to effectively integrate and share data from different biobanks. This data island phenomenon hinders the seamless connection and sharing of data between biobanks, and also limits the full utilization and mining potential of data. Therefore, a system based on sample library data processing is proposed. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the data in the biological sample library in the prior art is often stored in different platform ports in a scattered manner. The differences in data formats between different ports make it difficult to integrate and share the data between biological sample libraries, resulting in difficulty in achieving seamless docking and sharing of data between different sample libraries, which limits the full utilization of data and the potential for mining. A system for processing sample library data is proposed to solve the problem that the data in the biological sample library is often stored in different platform ports in a scattered manner. The differences in data formats between different ports make it difficult to integrate and share the data between biological sample libraries, which limits the full utilization of data and the potential for mining.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A system for processing sample library data, comprising:

[0007] Data collection and import module: responsible for collecting data from different port biobanks and importing them into the system;

[0008] Data standardization and conversion module: responsible for converting the data collected from different ports into the system standard format;

[0009] Data storage and management module: responsible for storing standardized data and providing data retrieval and management functions;

[0010] Data interface and sharing module: responsible for providing a unified API interface for data sharing between biobank samples at different ports;

[0011] Data integration and analysis module: responsible for integrating data extracted from different ports and performing analysis and mining;

[0012] The data collection and import module is responsible for collecting raw data from biobanks of various ports (such as hospitals, scientific research institutions, biobanks, etc.). The data includes basic information of samples (such as sample type, source, collection time, etc.), experimental data (such as gene sequence, protein expression level, etc.) and storage and management information of samples (such as storage location, temperature, humidity, etc.). The collected data is imported into the system and verified to ensure that the data is not lost or damaged. The data standardization and conversion module is responsible for cleaning the data (removing duplicates, null values ​​or abnormal values ​​in the data to ensure data quality), standardizing (unifying data in different sample libraries to the same standard) and converting (converting data from the original format to a unified target format). The data storage and management module is responsible for storing the standardized data in a planned manner. to the system, ensure the integrity and availability of data, encrypt the stored data, ensure the security of sensitive data, and prevent unauthorized access. The data interface and sharing module is responsible for designing a unified interface based on the interface protocol, data transmission format, and data exchange specifications, and formulating data sharing strategies based on business needs and data security requirements, clarifying the scope, method, and authority of data sharing, and implementing corresponding access control mechanisms, such as user authentication, authority management, and data encryption. The data integration and analysis module is responsible for integrating data from different ports, merging scattered data into a unified database through unified field mapping and relationship processing, selecting appropriate analysis methods to conduct in-depth analysis of data based on business needs and analysis purposes, and using data mining technology to explore potential associations, laws, or patterns.

[0013] The above technical solution further includes:

[0014] Preferably, the specific process of collecting data from biobanks at different ports is:

[0015] Connection and interface configuration: connect to biobanks at different ports through APIs and ensure data legitimacy through security authentication;

[0016] Data collection: The import of historical data uses a scheduled task to batch obtain data from the biological sample library. There are n data sources, and the data sets of each data source are recorded as D1, D2, ..., D n , each data set contains different types of information, such as sample number, collection time, experimental data, etc., from the data source D i Get dataset D i ={(ID 1, data1),(ID 2, data2),...,(ID k, data k )}, where each ID represents a unique identifier of a sample, data represents experimental data, and real-time data is collected by the system using polling and event triggering to obtain incremental data in real time.

[0017] Preferably, the specific process of importing into the system is:

[0018] Map the collected data to the database table structure in the system, including field matching, unit conversion, etc. Use batch import tools to import the processed data into the database. You can use the ETL (extract, transform, load) process to ensure that the data is imported effectively, and the system can handle real-time data transmission and store the data in the system immediately;

[0019] After the data is imported, the system performs integrity verification (confirming whether the imported data is complete and accurate) and consistency check (ensuring that all data is correctly mapped into the system to avoid duplication, loss or inconsistent formats).

[0020] Preferably, the specific process of converting the data collected from different ports into a system standard format is as follows:

[0021] Data cleaning: Check and remove identical or duplicate records. Duplicate data may be caused by multiple collections, import errors, etc. Identify and process null values ​​in the data. Null values ​​may be caused by measurement failure or missing data. Identify and remove outliers by setting the Z-score analysis method. Outliers may be caused by equipment failure, input errors, etc.

[0022] Data standardization: Use format conversion functions to convert different data formats into a unified data format, use unit conversion functions to convert different data units into a unified data unit, use data standardization methods to ensure data value consistency, and use metadata management methods to ensure data definition consistency;

[0023] Data conversion: According to the predetermined mapping rules, the source data fields are mapped to the target data fields one by one to ensure that the data can be accurately converted to the target format, and executable code is generated in SQL language to convert the data according to the defined data mapping rules.

[0024] Preferably, the specific process of storing the standardized data is:

[0025] According to the data type, usage scenario and system requirements, first plan the data storage structure, including the design of data tables, the establishment of indexes, the division of data partitions, etc., then determine the storage media and database selection to ensure the efficiency and scalability of data storage, and store the standardized data in the system according to the planned storage method. For large data sets, distributed storage methods can be used to improve storage capacity and query performance, and sensitive data can be encrypted and stored to ensure data security.

[0026] Preferably, the specific process of providing data retrieval and management functions includes:

[0027] The user submits a query request through the API interface. After receiving the query request, the system parses the query conditions. Based on the parsed query conditions, the system constructs the corresponding database query statement and submits the constructed query statement to the database for execution. The database retrieves data records that meet the conditions based on the query statement. The system encapsulates the retrieved data records and returns them to the user through the API interface.

[0028] The system uses encryption algorithms to encrypt sensitive data. Suppose the stored data D is encrypted and the encryption process using encryption algorithm Enc is expressed as: enc =Enc(D,K), where K is the encryption key, D enc is the encrypted data. The decryption process using the decryption algorithm Dec is expressed as: D = Dec (D enc ,K).

[0029] Preferably, the specific process of providing a unified API interface is:

[0030] Demand analysis: According to business needs, determine the specific functions that the API interface needs to provide, such as data query, data import, data export, etc.;

[0031] Interface design: select appropriate interface protocols, design reasonable API endpoints, create data models, define the structure, relationship and attributes of data, and clarify the parameter specifications of the API interface, including request parameters and response parameters;

[0032] Interface implementation: According to the interface design document, write corresponding code to implement the functions of the API interface, implement the data format conversion function in the API interface, and design a unified authentication and authorization mechanism to ensure that only verified users or applications can access the API interface;

[0033] Interface testing: Write comprehensive test cases, including normal and abnormal situations, as well as large data volumes and high concurrency situations, and use test tools to execute test cases to verify whether the functions and performance of the API interface meet expectations;

[0034] Interface publishing: Write detailed interface documentation, including interface function introduction, parameter description, return result examples, etc., and publish the interface documentation to the internal document management system or external platform.

[0035] Preferably, the specific process of integrating the data extracted from different ports is:

[0036] Field mapping and data alignment: Identify data fields from different ports and establish mapping relationships between data fields. Based on the mapping relationships, convert source data fields into a format compatible with the target fields.

[0037] Data relationship processing: Analyze the associations between different data sources, such as primary and foreign key relationships, business logic relationships, etc. Based on the identified relationships, establish the corresponding table structure and associations in the integrated database;

[0038] Data merging and integration: Merge data from different ports into the integrated database according to mapping relationships and business logic, and perform further integration processing to eliminate data redundancy, resolve data conflicts, etc.

[0039] The present invention has the following beneficial effects:

[0040] 1. In the present invention, raw data can be collected from multiple port biobanks through the data collection and import module, and the data can be cleaned, standardized and converted through the data standardization and conversion module, thereby solving the problem of data format differences between different ports. The data integration and analysis module further merges the scattered data into a unified database, so that data can be seamlessly connected and shared between different sample libraries.

[0041] 2. In the present invention, the data integration and analysis module can not only integrate data from different ports, but also select appropriate analysis methods to conduct in-depth analysis of the data according to business needs and analysis purposes. Using data mining technology, the data integration and analysis module can mine potential associations, rules or patterns, thereby improving the utilization efficiency and mining potential of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a system architecture diagram of a sample library data processing system proposed by the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, a system for processing sample library data includes:

[0045] Data collection and import module: responsible for collecting data from different port biobanks and importing them into the system;

[0046] Data standardization and conversion module: responsible for converting the data collected from different ports into the system standard format;

[0047] Data storage and management module: responsible for storing standardized data and providing data retrieval and management functions;

[0048] Data interface and sharing module: responsible for providing a unified API interface for data sharing between biobank samples at different ports;

[0049] Data integration and analysis module: responsible for integrating data extracted from different ports and performing analysis and mining;

[0050] The data collection and import module is responsible for collecting raw data from various port biobanks, importing the collected data into the system and verifying the data. The data standardization and conversion module is responsible for cleaning, standardizing and converting the data. The data storage and management module is responsible for storing the standardized data in the system in a planned manner and encrypting the stored data. The data interface and sharing module is responsible for designing a unified interface based on the interface protocol, data transmission format and data exchange specifications, and formulating data sharing strategies and implementing corresponding access control mechanisms based on business needs and data security requirements. The data integration and analysis module is responsible for integrating data from different ports, merging scattered data into a unified database, selecting appropriate analysis methods based on business needs and analysis purposes to conduct in-depth analysis of the data, and using data mining technology to explore potential associations, laws or patterns.

[0051] In the embodiment of the present invention, selecting appropriate analysis methods to conduct in-depth analysis of data includes descriptive statistical analysis and correlation analysis. Descriptive statistical analysis performs basic statistical analysis on the data to help understand the basic situation of the data. For example, the calculation of the data set D = {d1, d2, ..., d n}, Correlation analysis analyzes the linear relationship between variables by calculating the Pearson correlation coefficient. For example, for two variables X and Y, the Pearson correlation coefficient r is defined as: Among them, μ X and μ Y are the means of X and Y respectively. And use data mining technology to mine potential associations, rules or patterns, such as clustering data through K-means algorithm. Assume that the data set is D = {d1, d2, ..., d n}, the number of clusters is k, then the objective minimization of the K-means algorithm is expressed as: Among them, C i is the i-th cluster, μ i is the center of the cluster, ||d j -μ i || is the distance between the data point and the cluster center.

[0052] In one embodiment, the specific process of collecting data from different port biobanks is as follows:

[0053] Connection and interface configuration: connect to biobanks at different ports through APIs and ensure data legitimacy through security authentication;

[0054] Data collection: The import of historical data uses a scheduled task to batch obtain data from the biological sample library. There are n data sources, and the data sets of each data source are recorded as D1, D2, ..., D n , each dataset contains different types of information from data source D i Get dataset D i ={(ID 1, data1),(ID 2, data2),...,(ID k, data k )}, where each ID represents a unique identifier of a sample, data represents experimental data, and real-time data is collected by the system using polling and event triggering to obtain incremental data in real time.

[0055] In the embodiment of the present invention, the security authentication used to ensure the legitimacy of data includes OAuth authentication, API key authentication and basic identity authentication. The steps of OAuth authentication are: first, the system requests an authorization code from the authentication service of the biobank. The system will guide the user to authorize, use the authorization code to request an access token, and attach the obtained access token to the header of each API request for identity authentication. The steps of API key authentication are: obtain the API key assigned by the system, and use the API key as part of the request, usually passed in the request header. The steps of basic identity authentication are: use a combination of username and password for basic authentication.

[0056] In one embodiment, the specific process of importing into the system is:

[0057] Map the collected data to the database table structure in the system, use the batch import tool to import the processed data into the database, and the system can handle real-time data transmission and store the data in the system immediately;

[0058] After completing the data import, the system performs integrity verification and consistency check.

[0059] In the embodiment of the present invention, for real-time data transmission, the real-time data set is assumed to be D realtime , each collected data is an incremental data set D new , obtained by comparing timestamps: D new ={d i │t i >t last}, where t last is the timestamp of the last import time. Integrity verification ensures that all fields are filled and not missing. Suppose the field set of a data record is {f1,f2,...,f m}, then the integrity check formula is: That is, each field f j Must be non-empty. The consistency check ensures that the logical relationship between the data is consistent, for example, the sample ID must be unique. Assume that the sample ID set is {SampleID1,SampleID2,...,SampleID n}, the consistency check formula is: That is, each sample ID must be unique.

[0060] In one embodiment, the specific process of converting the data collected from different ports into a system standard format is as follows:

[0061] Data cleaning: Check and remove identical or duplicate records, identify and process null values ​​in the data, and identify and remove outliers by setting the Z-score analysis method;

[0062] Data standardization: Use format conversion functions to convert different data formats into a unified data format, use unit conversion functions to convert different data units into a unified data unit, use data standardization methods to ensure data value consistency, and use metadata management methods to ensure data definition consistency;

[0063] Data conversion: According to the predetermined mapping rules, the source data fields are mapped one by one with the target data fields, and the executable code is generated in SQL language to convert the data according to the defined data mapping rules.

[0064] In the embodiment of the present invention, the conversion function of the data format is assumed to be f convert (d i ), the data after format standardization is d standardized =f convert (d i ). Assume the original unit of the data is U raw and the target unit is U target , then the unit conversion formula is: in, is the unit conversion constant. Assume the original field is f src and the target field is f dst , then the mapping rule is map(f src )→f dst , then the data conversion formula is: f dst =map(f src ).

[0065] In one embodiment, the specific process of storing the standardized data is as follows:

[0066] According to the data type, usage scenario and system requirements, first plan the data storage structure, then determine the storage media and database selection, and store the standardized data into the system according to the planned storage method.

[0067] In the embodiment of the present invention, a hierarchical storage structure is designed according to the types of data, such as biological sample data, experimental data, clinical data, etc., and the usage scenarios of the data, such as query, analysis, backup, etc. Assume that there are N types of data, each type of data C i A storage area S will be allocated i , then: S i ={d j ∈C i │j=1,2,...,m}, where d j is a data item, and m is the number of entries of this type of data.

[0068] In one embodiment, the specific process of providing data retrieval and management functions includes:

[0069] The user submits a query request through the API interface. After receiving the query request, the system parses the query conditions. Based on the parsed query conditions, the system constructs the corresponding database query statement and submits the constructed query statement to the database for execution. The database retrieves data records that meet the conditions based on the query statement. The system encapsulates the retrieved data records and returns them to the user through the API interface.

[0070] The system uses encryption algorithms to encrypt sensitive data. Suppose the stored data D is encrypted and the encryption process using encryption algorithm Enc is expressed as: enc =Enc(D,K), where K is the encryption key, D enc is the encrypted data. The decryption process using the decryption algorithm Dec is expressed as: D = Dec (D enc ,K).

[0071] In one embodiment, the specific process of providing a unified API interface is:

[0072] Demand analysis: Determine the specific functions that the API interface needs to provide based on business needs;

[0073] Interface design: select appropriate interface protocols, design reasonable API endpoints, create data models, define the structure, relationship and attributes of data, and clarify the parameter specifications of API interfaces;

[0074] Interface implementation: According to the interface design document, write corresponding code to implement the functions of the API interface, implement the data format conversion function in the API interface, and design a unified authentication and authorization mechanism;

[0075] Interface testing: write comprehensive test cases and execute them using testing tools;

[0076] Interface publishing: Write detailed interface documentation and publish the interface documentation to the internal document management system or external platform.

[0077] In one embodiment, the specific process of integrating data extracted from different ports is as follows:

[0078] Field mapping and data alignment: Identify data fields from different ports and establish mapping relationships between data fields. Based on the mapping relationships, convert source data fields into a format compatible with the target fields.

[0079] Data relationship processing: Analyze the associations between different data sources, and based on the identified relationships, establish corresponding table structures and associations in the integrated database;

[0080] Data merging and integration: Merge data from different ports into the integrated database according to mapping relationships and business logic, and perform further integration processing.

[0081] In the embodiment of the present invention, the purpose of field mapping is to identify the same or similar fields in different data sources and establish mapping relationships between them. In implementation, it is assumed that there are N source data field sets F = {F1, F2, ..., F N}, and the target field set T in the target database = {T1, T2, ..., T M}. The mapping relationship is Mapping:F i →T j , where F i is the source data field, T j is the target data field. The purpose of mapping is to convert F i The fields in T are converted to j Compatible formats. Assume that there are n data sources, and the data sets of each data source are recorded as D1, D2, ..., D n The data in these data sources are merged according to the mapping rules, and the merged data set is D merged The merging process is expressed as:

[0082] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A system for processing sample library data, characterized in that: include: Data collection and import module; Responsible for collecting data from different port biobanks and importing them into the system; Data standardization and conversion module: responsible for converting the data collected from different ports into the system standard format; Data storage and management module: responsible for storing standardized data and providing data retrieval and management functions; Data interface and sharing module: responsible for providing a unified API interface for data sharing between biobank samples at different ports; Data integration and analysis module: responsible for integrating data extracted from different ports and performing analysis and mining; The data collection and import module is responsible for collecting raw data from biobanks of various ports, importing the collected data into the system and verifying the data; the data standardization and conversion module is responsible for cleaning, standardizing and converting the data; the data storage and management module is responsible for storing the standardized data in the system in a planned manner and encrypting the stored data; the data interface and sharing module is responsible for designing a unified interface based on interface protocols, data transmission formats and data exchange specifications, and formulating data sharing strategies and implementing corresponding access control mechanisms based on business needs and data security requirements; the data integration and analysis module is responsible for integrating data from different ports, merging scattered data into a unified database, selecting appropriate analysis methods based on business needs and analysis purposes to conduct in-depth analysis of the data, and using data mining technology to mine potential associations, laws or patterns.

2. A system for processing sample library data according to claim 1, characterized in that: The specific process of collecting data from different port biobanks is as follows: Connection and interface configuration: connect to biobanks at different ports through APIs and ensure data legitimacy through security authentication; Data collection: The import of historical data uses a scheduled task to batch obtain data from the biological sample library. There are n data sources, and the data sets of each data source are recorded as D1, D2, ..., D n , each dataset contains different types of information from data source D i Get dataset D i ={(ID 1, data1),(ID 2, data2),...,(ID k, data k )}, where each ID represents a unique identifier of a sample, data represents experimental data, and real-time data is collected by the system using polling and event triggering to obtain incremental data in real time.

3. A system for processing sample library data according to claim 1, characterized in that: The specific process of importing into the system is: Map the collected data to the database table structure in the system, use the batch import tool to import the processed data into the database, and the system can handle real-time data transmission and store the data in the system immediately; After completing the data import, the system performs integrity verification and consistency check.

4. A system for processing sample library data according to claim 1, characterized in that: The specific process of converting the data collected from different ports into the system standard format is as follows: Data cleaning: Check and remove identical or duplicate records, identify and process null values ​​in the data, and identify and remove outliers by setting the Z-score analysis method; Data standardization: Use format conversion functions to convert different data formats into a unified data format, use unit conversion functions to convert different data units into a unified data unit, use data standardization methods to ensure data value consistency, and use metadata management methods to ensure data definition consistency; Data conversion: According to the predetermined mapping rules, the source data fields are mapped one by one with the target data fields, and executable code is generated in SQL language to convert the data according to the defined data mapping rules.

5. The system for processing sample library data according to claim 1, characterized in that: The specific process of storing the standardized data is as follows: According to the data type, usage scenario and system requirements, first plan the data storage structure, then determine the storage media and database selection, and store the standardized data into the system according to the planned storage method.

6. A system for processing sample library data according to claim 1, characterized in that: The specific process of providing data retrieval and management functions includes: The user submits a query request through the API interface. After receiving the query request, the system parses the query conditions. Based on the parsed query conditions, the system constructs the corresponding database query statement and submits the constructed query statement to the database for execution. The database retrieves data records that meet the conditions based on the query statement. The system encapsulates the retrieved data records and returns them to the user through the API interface. The system uses encryption algorithms to encrypt sensitive data. Suppose the stored data D is encrypted and the encryption process using encryption algorithm Enc is expressed as: enc =Enc(D,K), where K is the encryption key, D enc is the encrypted data. The decryption process using the decryption algorithm Dec is expressed as: D = Dec (D enc ,K).

7. A system for processing sample library data according to claim 1, characterized in that: The specific process of providing a unified API interface is as follows: Demand analysis: Determine the specific functions that the API interface needs to provide based on business needs; Interface design: select appropriate interface protocols, design reasonable API endpoints, create data models, define the structure, relationship and attributes of data, and clarify the parameter specifications of API interfaces; Interface implementation: According to the interface design document, write corresponding code to implement the functions of the API interface, implement the data format conversion function in the API interface, and design a unified authentication and authorization mechanism; Interface testing: write comprehensive test cases and execute them using testing tools; Interface publishing: Write detailed interface documentation and publish the interface documentation to the internal document management system or external platform.

8. The system for processing sample library data according to claim 1, characterized in that: The specific process of integrating the data extracted from different ports is as follows: Field mapping and data alignment: Identify data fields from different ports and establish mapping relationships between data fields. Based on the mapping relationships, convert source data fields into a format compatible with the target fields. Data relationship processing: Analyze the associations between different data sources, and based on the identified relationships, establish corresponding table structures and associations in the integrated database; Data merging and integration: Merge data from different ports into the integrated database according to mapping relationships and business logic, and perform further integration processing.

Images