Scientific research platform system based on big data

Abstract

The invention discloses a scientific research platform system based on big data, and the system comprises a data extraction module which is used for carrying out the extraction of mode data and instance data through employing an incremental extraction algorithm according to a processing scheme of the invention; the data integration module is used for uniformly mapping the mode data and the instance data and converting the mode data and the instance data into a single result set; the data protocol module is used for eliminating similar duplicate records after data integration; the data loading module is used for loading the data set after the data reduction to one or more target databases; and the end point data storage module is used for storing the loaded data and metadata. Through the processing scheme disclosed by the invention, various data are converged, unified rule data cleaning, integrated aggregation data expression and established model data analysis are performed on the extracted data, and reasonable interpretation visualization is finally realized through algorithms such as text mining, deep learning, clustering, association rules and logistic regression.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a scientific research platform system based on big data. Background Art

[0002] In the actual scientific research process, hospitals and clinicians often face many realistic and urgent problems. The existence of these problems and bottlenecks greatly restricts the entire process of clinical research.

[0003] 1. Manual and electronic data collection in parallel The questionnaires for each period of the birth cohort study were different. Some of them needed to be manually transcribed from paper documents into electronic documents. The quality of the transcription process could not be guaranteed and the transcription took a certain amount of time. Other documents were electronic documents exported through WJX, provincial platforms or from hospital systems. They needed to be processed by a dedicated person when the data was aggregated. The data collection for the same patient might be in multiple situations, which made it inconvenient for project-related personnel to directly browse, control the quality and use the data in a unified manner.

[0004] 2. Multi-caliber data storage method During the data collection process of the cohort study, it was found that the storage caliber of the questionnaire database was partially different, that is, the database was scattered in different places in different forms, which made it inconvenient to count the questionnaire follow-up status of each subject at different periods.

[0005] 3. Complex data query logic Medical research is different from scientific research in other fields. Patient data has complex time and event relationships. In data acquisition, we often encounter the dilemma of "there are data, but we can't find them". In the end, we can only rely on manual judgment one by one. There is a lack of professional scientific research data search systems to support complex logical queries and operations.

[0006] 4. Difficulty in phased verification of medical research A medical study often lasts for several years or even longer, and during the entire research process, phased research verification is rarely carried out to ensure the feasibility of the entire research plan and the quality of the research results.

[0007] 5. Poor availability of research data The quality of research data is a key factor in determining the quality of research results. If the data quality is poor, the conclusions drawn will definitely be unreliable. The current medical research model, which mainly relies on manual methods of free text data, can only rely on the researchers' own level to ensure the quality of data. Even if there is third-party supervision, it will be difficult to ensure data quality due to professional, communication, time and other factors. At present, most studies have problems such as difficulty in data traceability, low quality of data available for research, and lack of effective link monitoring in the research process, making it impossible to objectively evaluate the quality of research data.

[0008] The above method has several disadvantages in actual use, and thus fails to achieve the best effect. The disadvantages can be summarized as follows: 1. Data is collected repeatedly and the utilization rate is low Most medical institutions will encounter this problem in the later data utilization process, which is mainly manifested in the fluid separation of clinical and scientific research data, that is, clinical medical record writing only serves medical record management, while scientific research data needs to be collected separately. Important clinical information about patients in patient medical records cannot be used directly due to their unstructured or semi-structured storage methods. They can only be transcribed or excerpted manually, which greatly increases the time cost of scientific research data collection.

[0009] 2. Data storage is scattered and structured differently Scientific research data often comes from multiple business systems depending on the purpose of the research topic, and may even widely involve patient follow-up, CRF forms, biological samples, omics analysis and other data. In the actual data collection and sorting process, doctors often have to switch back and forth between multiple systems and manually correlate and check the data from multiple systems, which significantly increases the cost of errors.

[0010] 3. Lack of unified monitoring of data quality and integrity Since clinical research has high requirements for data quality and integrity, but lacks an effective and targeted platform for unified monitoring and management of data in the early stages, it often leads to the inability to carry out the next step of clinical research due to the missing or low completeness of a key variable in the final stage of statistical analysis.

[0011] 4. The follow-up date is calculated manually, and the follow-up process is time-consuming and laborious For research project follow-up, currently follow-up staff are required to manually calculate the follow-up date and then remind the participants via phone calls and text messages. The workload of follow-up staff is heavy and the process is time-consuming and laborious.

[0012] 5. Difficulty in sharing research data Within medical research institutions, such as hospitals and departments, and between different research institutions, different studies are carried out independently. Even for the same subject or disease, research data can rarely be reused. Existing research data becomes "dead data" after playing a role once. The high cost of collecting data will not be converted into scientific research output again, and the input-output ratio is high. Data sharing between different studies can solve this problem, but the data needs to meet certain requirements, and current research data cannot reach this level.

[0013] It can be seen that the above existing systems still have inconveniences and defects in use, and are in urgent need of further improvement. How to create a new scientific research platform based on big data has become a goal that the industry urgently needs to improve. Summary of the invention

[0014] In view of this, the embodiments of the present disclosure provide a scientific research platform system based on big data, which at least partially solves the problems existing in the prior art.

[0015] In a first aspect, the present disclosure provides a scientific research platform system based on big data, the system comprising: A data extraction module, used to extract pattern data and instance data by using an incremental extraction algorithm; A data integration module, used for uniformly mapping the pattern data and the instance data and converting them into a single result set; Data reduction module, used to eliminate similar duplicate records after data integration; A data loading module is used to load the data set after data reduction into one or more target databases; The end point data storage module is used to store the loaded data and metadata.

[0016] According to a specific implementation of the embodiment of the present disclosure, the data integration module also includes: The data verification module is used to verify the consistency of data.

[0017] According to a specific implementation of the embodiment of the present disclosure, the system further includes: Universal data access interface for cross-platform data access.

[0018] According to a specific implementation of the embodiment of the present disclosure, the system further includes: System management module, the system management module includes: workflow management module, job scheduling management module, log management module, system monitoring module, data recovery / backup module; Among them, the workflow management module is used to customize jobs and ETL steps: The job scheduling management module is used to allocate the running time of ETL jobs and steps; The log management module is used to provide a memorandum of events that occurred during the ETL process; The system monitoring module is used to monitor the dynamic operation of the ETL process, report runtime errors, system anomalies and efficiency bottlenecks; The data recovery / backup module is used to provide backup and recovery of data at each stage of the ETL process.

[0019] According to a specific implementation of the embodiment of the present disclosure, the system further includes: Program access interface, used to integrate data ETL with its subsystems through the interface.

[0020] According to a specific implementation of the embodiment of the present disclosure, the system further includes: The feature extraction module is used to segment the pattern data and instance data and perform analysis and statistics.

[0021] According to a specific implementation of the embodiment of the present disclosure, the system further includes: The analysis module is used to obtain the correlation between data through data mining and machine learning.

[0022] The big data-based scientific research platform system in the disclosed embodiment gathers hospital clinical, imaging, biological and other data, and performs unified rule data cleaning on the extracted data, integrates the aggregated data expression, establishes model data analysis, and finally realizes reasonable interpretation and visualization through text mining, deep learning, clustering, association rules, logistic regression and other algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0024] Figure 1 A schematic diagram of a scientific research platform system based on big data provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of the structure of a scientific research platform system based on big data provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of the relationship between a big data research platform and informatization and data flow provided in an embodiment of the present disclosure; and Figure 4 A schematic diagram of a birth cohort topic flow chart provided in an embodiment of the present disclosure; Figure 5 A data aggregation schematic diagram provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0026] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0027] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement a device and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0028] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0029] The scientific research platform based on big data has gradually shown its important position and advantages in the process of building research hospitals, and has also become a new hot spot for new clinical research models based on big data in recent years. The big data platform is based on Hadoop technology, which realizes the standardized storage of massive heterogeneous data, including unstructured medical record data and large-scale imaging data, and realizes the standardized processing of data through technologies such as natural language processing. At the same time, the collected medical data provides massive training and verification sets for methods such as deep learning, which will greatly promote the intelligent analysis of medical data, especially the natural language processing of medical record data and high-performance analysis of medical images.

[0030] The data content covers electronic medical records, medical images, sample libraries, gene sequencing, laboratory data, etc. The data form, data source and data storage will grow explosively, and data analysis and model calculation require a large amount of distributed computing resources. Therefore, it is necessary to establish a platform for big data aggregation, secure storage and analysis, and provide standardized big data services for various PIs.

[0031] The embodiment of the present invention provides a scientific research platform system based on big data. Through the construction and implementation of big data data processing and data analysis platform, the time cost of PI using data can be reduced, plug-and-play data services can be enjoyed, the implementation cycle of clinical practice and clinical research can be significantly shortened, and the transformation of results can be accelerated; the application of big data analysis technology and data mining algorithms can assist clinicians in conceiving scientific research ideas and proposing causal hypotheses to the greatest extent; clinical trials based on large samples and full data can quickly complete problem verification and generate statistical results, reducing scientific research costs; the construction of a large number of algorithm model libraries and statistical model libraries can precipitate the laboratory's knowledge assets for continuous learning and utilization; with the help of high-level data security measures, the experimental platform can be effectively guaranteed to provide high-quality data services for the majority of clinical researchers.

[0032] Figure 1 A schematic diagram of a scientific research platform system based on big data provided in an embodiment of the present disclosure.

[0033] The embodiment of the present invention provides a scientific research platform system based on big data, and the system includes: a data extraction module, a data integration module, a data reduction module, a data loading module and an end point data storage module.

[0034] Among them, the data extraction module is used to extract pattern data and instance data by using an incremental extraction algorithm; the data integration module is used to uniformly map the pattern data and instance data and convert them into a single result set; the data reduction module is used to eliminate similar duplicate records after data integration; the data loading module is used to load the data set after data reduction into one or more destination databases; the terminal data storage module is used to store the loaded data and metadata.

[0035] In the embodiment of the present invention, the data integration module further includes: a data verification module for performing consistency verification of data.

[0036] In the embodiment of the present invention, the system further includes: a universal data access interface for accessing data across platforms.

[0037] In an embodiment of the present invention, the system also includes: a system management module, which includes: a workflow management module, a job scheduling management module, a log management module, a system monitoring module, and a data recovery / backup module; wherein the workflow management module is used for customizing jobs and ETL steps; the job scheduling management module is used for allocating the running time of ETL jobs and steps; the log management module is used for providing a memorandum of events occurring during the ETL process; the system monitoring module is used for monitoring the dynamic running status of the ETL process, reporting runtime errors, system anomalies and efficiency bottlenecks; the data recovery / backup module is used for providing backup and recovery of data at each stage of the ETL process.

[0038] In the embodiment of the present invention, the system further comprises: a program access interface, which is used to integrate the data ETL with its subsystem through the interface.

[0039] In the embodiment of the present invention, the system further includes: a feature extraction module for performing word segmentation processing on the pattern data and the instance data, and performing analysis and statistics.

[0040] In an embodiment of the present invention, the system further includes: an analysis module, which is used to obtain the correlation relationship between data through data mining and machine learning.

[0041] More specifically, data integration based on big data, such as Figure 2 , Figure 3 As shown, specifically including: The ETL structural model design includes data flow and metadata flow from the source to the destination database. The data flow is the flow of instance data, which can become data integration, and it is the ultimate goal of data ETL; part of the metadata flow is the pattern data, and the other part is the logic specification formulation information, which is used to determine the data transformation steps that need to be performed in the cleaning process. This part of metadata mainly plays an auxiliary role in the ETL process. Using a universal data access interface to shield the differences between various data sources, with data cleaning as the main purpose, provides an effective solution for eliminating pattern conflicts and data conflicts among multiple data sources. The functions and processes of each part are detailed as follows:

[0042] Heterogeneous multiple data sources: Support multiple data sources, such as ODBC data sources, various relational database data sources, data in various file formats, etc. Universal data access interface: This interface can access data across platforms and supports establishing connections between different types of data sources. It can shield the differences between various data sources and provide a unified data view for subsequent work; Data extraction: including pattern data and instance data extraction. First, pattern information is extracted from the data source, analyzed manually or with relevant intelligent algorithms, and instance data extraction strategies are obtained, which are stored in the knowledge base for use when loading data. This process uses an incremental extraction algorithm to avoid time-consuming repeated scanning of large multi-source data.

[0043] Data integration: After data extraction, multiple models and multiple instance data sets can be obtained. However, the data warehouse in the platform requires integrated and semantically consistent data, so multiple models and data sets must be uniformly mapped and converted into a single result set. In this process, various data sources should first be converted to obtain a data structure with a unified format; then data standardization should be performed to eliminate inconsistent abbreviations and abbreviations; then data consistency verification should be performed; various errors in the content should be corrected, and finally the processed data should be stored in a temporary area as an intermediate result, waiting for further cleaning;

[0044] Data reduction: After data integration, the data set still contains many similar duplicate records, which will seriously affect the consistency of data semantics and must be eliminated. In this process, the data set is first matched to find duplicate anomalies, and then processed according to the matching results, deleting some records or merging multiple records into a more complete information record;

[0045] Data loading: This process solves the problems of pattern implementation and data loading timing. It can selectively load the cleaned data set into one or more destination databases, allowing manual intervention and providing powerful error reporting, system logging, data backup and recovery functions. End point data storage: provides a storage place for data and metadata, which is the end point of ETL, and here is the data warehouse; System management: The platform has established a set of powerful auxiliary management tools based on the Kafka messaging mechanism. Among them, workflow management is mainly responsible for the customization of jobs and ETL steps; job scheduling management is responsible for allocating the running time of ETL jobs and steps; log management provides a memorandum of events that occurred during the ETL process; system monitoring is responsible for monitoring the dynamic operation of the ETL process, reporting runtime errors, system anomalies and efficiency bottlenecks; data recovery / backup, as the name implies, provides backup and recovery functions at each stage of the ETL process;

[0046] Programming Access Interface (API): Provides a powerful interface to the outside world, so that data ETL can be better integrated with its subsystems. In order to achieve better interactivity and scalability, ETL provides a descriptive language similar to SQL to facilitate the customization of efficient data processing processes.

[0047] ETL is the most important step in building a data warehouse. It extracts the required data from the data source, cleans the data, and finally loads the data into the data warehouse according to the pre-defined data warehouse model.

[0048] Resource management, including: The unified resource monitoring and management system can effectively manage and view the resource composition, resource usage, and resource-related KPI analysis of the big data resource center, and effectively audit resource consumption. It also provides data verification functions, monitors daily data incremental updates and the number of patients with synchronization failures, verifies data integrity in real time, and guarantees data quality to the greatest extent possible.

[0049] 1. Statistics The data content covers electronic medical records, operational data, disease resources, sample libraries, gene sequencing data, stem cells (IPS), medical images, laboratory data, etc. Through document data coverage analysis, we can intuitively understand the data composition and construction progress of the big data resource center.

[0050] 2. Patient distribution Understanding the distribution of resources can not only help management to keep abreast of the storage status of resources in the library, but also help clinical staff to grasp the scope of available data at any time and provide corresponding indicator references for scientific research sample screening.

[0051] 3. Disease distribution The main diagnoses of people discharged from hospital are statistically analyzed, and big data word cloud technology is applied to realize text data mining. The frequency of each diagnosis is displayed to help patients clearly understand the distribution of their diseases.

[0052] 4. Symptom word cloud distribution The symptom-related fields in the patient's medical record text are segmented and processed, and analyzed and counted using big data technology, allowing clinical researchers to intuitively and clearly see the distribution of symptom manifestations of the studied group.

[0053] 5. Disease symptom analysis Using data mining and machine learning techniques, we continuously learn the relationship between patient-related diseases and symptoms from patient medical records and domestic and international literature standards, helping doctors identify the common symptoms of a certain disease and the distribution of diseases corresponding to the symptoms.

[0054] 6. Data verification Data verification and data quality reports will be dynamically compiled based on the daily data extraction situation, including the amount of synchronously updated data, the amount of data that failed to be extracted, the reasons for failure, and the degree of data completeness of patient medical records, tests, examinations, and imaging data.

[0055] The present invention is described in detail below by way of examples.

[0056] 1. Birth Cohort Project Process like Figure 4 As shown in the figure, the birth cohort needs to collect the process examination information of pregnant women from 12 weeks of pregnancy to delivery, and collect samples during the same period of pregnancy and store them in the sample library. Based on the known cohort research process and data collection scope, a cohort research information platform is established. The specific platform construction process is analyzed as follows:

[0057] (1) The historically collected Excel data, provincial platform data, and existing in-hospital examination data were processed, integrated, and extracted into the birth cohort research database.

[0058] (2) Through a unified data collection platform, questionnaire information on the birth cohort process, including data from early pregnancy, mid-pregnancy, late pregnancy and different periods after the birth of the child, is collected and summarized into a unified birth cohort research database through mobile terminals, WeChat or PC terminals.

[0059] (3) By integrating with the in-hospital system data center, the clinical information needed in the cohort study, such as prenatal examinations, test reports, prescription information, etc., and the information on children's in-hospital examinations, medical treatment and medication information, etc., can be summarized through inter-system integration.

[0060] (4) The data on the provincial platform will be imported regularly to achieve the complete data aggregation required for cohort studies and form a scientific research data center for birth cohort studies.

[0061] like Figure 5 As shown, based on the analysis of the process and data collection scope of the birth cohort study, it is necessary to realize the birth cohort study scientific research data center through the establishment of the information system and related technical services in Table 1 below.

[0062] The scientific research platform system based on big data proposed in this invention integrates and mines massive medical data including electronic medical records, tests, medical images, gene sequences, etc. through technologies such as big data processing, natural language segmentation, machine learning, and knowledge graphs, forming a complete patient-centered time series research resource library, and using data mining algorithms to achieve in-depth analysis and visualization of clinical data, assisting clinicians in conceiving scientific research ideas and proposing etiological hypotheses. Based on large-sample clinical trials, problem verification can be completed quickly and statistical results can be generated, reducing scientific research costs and improving the service quality of medical institutions and the efficiency of scientific research results transformation.

[0063] The big data research platform supports multiple data retrieval methods such as advanced retrieval, manual import and full-text retrieval. Clinicians can flexibly choose according to the specific business application scenarios, or freely combine queries. The population is the core of the entire business system, and the population is managed through operations such as adding, deleting, updating and merging. Six major applications have been expanded based on the population, including population analysis, research modeling, data export, data quality control, data mining and patient case views.

[0064] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A scientific research platform system based on big data, characterized in that: The system comprises: A data extraction module, used to extract pattern data and instance data by using an incremental extraction algorithm; A data integration module, used for uniformly mapping the pattern data and the instance data and converting them into a single result set; Data reduction module, used to eliminate similar duplicate records after data integration; A data loading module is used to load the data set after data reduction into one or more target databases; The end point data storage module is used to store the loaded data and metadata.

2. The scientific research platform system based on big data according to claim 1, characterized in that: The data integration module also includes: The data verification module is used to verify the consistency of data.

3. The scientific research platform system based on big data according to claim 1, characterized in that: The system further comprises: Universal data access interface for cross-platform data access.

4. The scientific research platform system based on big data according to claim 1, characterized in that: The system further comprises: System management module, the system management module includes: workflow management module, job scheduling management module, log management module, system monitoring module, data recovery / backup module; Among them, the workflow management module is used to customize jobs and ETL steps: The job scheduling management module is used to allocate the running time of ETL jobs and steps; The log management module is used to provide a memorandum of events that occurred during the ETL process; The system monitoring module is used to monitor the dynamic operation of the ETL process, report runtime errors, system anomalies and efficiency bottlenecks; The data recovery / backup module is used to provide backup and recovery of data at each stage of the ETL process.

5. The scientific research platform system based on big data according to claim 1, characterized in that: The system further comprises: Program access interface, used to integrate data ETL with its subsystems through the interface.

6. The scientific research platform system based on big data according to claim 1, characterized in that: The system further comprises: The feature extraction module is used to segment the pattern data and instance data and perform analysis and statistics.

7. The scientific research platform system based on big data according to claim 1, characterized in that: The system further comprises: The analysis module is used to obtain the correlation between data through data mining and machine learning.

Images