Big data-based laboratory data rapid processing and sharing method

Abstract

The application discloses a laboratory data rapid processing and sharing method based on big data, and relates to the technical field of medical information.The application can efficiently process the massive data generated by the laboratory by means of the big data storage architecture and parallel processing technology, can significantly shorten the data processing time, can meet the timeliness requirement of clinical rapid diagnosis, can use the big data analysis algorithm to mine data association, can extract potential rules such as disease trend and treatment effect from the test data, and can provide quantitative support for clinical diagnosis; the visual presentation mode can improve the test result interpretation efficiency, can facilitate doctors to quickly obtain key information, can build a safe sharing platform, can realize the rapid and convenient sharing of the laboratory data within the authorized range, can meet the requirement of the clinical rapid result acquisition, can meet the medical data compliance requirement, and can be conveniently extended based on the big data technology to adapt to the increase of the future data volume and the increase of the processing requirement.

Description

Technical Field

[0001] This invention relates to the field of medical information technology, and in particular to a method for rapid processing and sharing of laboratory data based on big data. Background Technology

[0002] Traditional data processing methods often rely on local servers and relatively isolated information systems, resulting in limited processing speed, low data sharing efficiency, and difficulty in meeting the needs of clinical departments for rapid access to test results. For example, Chinese patent application CN114242262A discloses a rapid medical research information processing system based on big data records. This system includes a central processing platform, whose input end is connected to the output end of an information input module. The central processing platform provides server support and, through a designed information optimization unit, optimizes the hierarchical data entry process after data aggregation. Simultaneously, the designed academic recommendation module and internal evaluation platform facilitate the rating and circulation of medical research information, extending its value and improving information data exchange in medical research and teaching. Furthermore, the designed filtering unit enables deduplication of fields in large amounts of data, and the retrieval of content fields allows for content analysis and classification, meeting overall usage needs.

[0003] However, although the aforementioned patents achieve hierarchical data entry and deduplication screening through a central processing platform and information optimization unit, the following problems still exist:

[0004] 1. The centralized data processing architecture has limited data sharding and parallel processing capabilities when dealing with the TB-level data generated by the laboratory every day. It cannot meet the second-level response requirements of emergency laboratory data, and processing delays are likely to occur, especially during peak traffic periods.

[0005] 2. The judgment rules cannot be dynamically adjusted according to the equipment type and historical data distribution, resulting in low accuracy in identifying abnormal data in scenarios involving new testing equipment or reagent changes; and the co-storage of high-time-sensitive emergency data and low-frequency scientific research data leads to increased read / write latency of critical data and insufficient utilization of storage resources.

[0006] 3. When medical institutions share test data, there is a lack of a unified sharing architecture and standard protocols, which limits the application scenarios of data and makes the data sharing process cumbersome and lacks automation. Summary of the Invention

[0007] The purpose of this invention is to provide a method for rapid processing and sharing of laboratory data based on big data, which significantly improves the processing speed of laboratory data and enables efficient data sharing across departments and institutions, thereby better supporting clinical decision-making and scientific research, and solving the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] Methods for rapid data processing and sharing in laboratory medicine based on big data include:

[0010] Laboratory data acquisition: Based on the constructed data acquisition interface, communication connections are established with various testing equipment to collect raw data generated by each testing equipment in the laboratory during the testing process in real time. At the same time, the collected raw data is preprocessed to form standardized laboratory data.

[0011] Rapid data processing: Based on a distributed storage database, data sharding and redundant storage strategies are adopted to distribute standardized laboratory data across various storage nodes. According to data analysis and sharing needs, format conversion and integration are performed to form a complete set of laboratory data.

[0012] Data analysis: Perform data analysis and feature extraction on the laboratory data set to uncover the potential value and patterns in the data, and present the analysis results in an intuitive form;

[0013] Data sharing: Build a data sharing platform and establish a data sharing link between the platform and the distributed storage database to achieve real-time data sharing; at the same time, establish data sharing standards and processes, clarify the scope, methods and responsibilities of data sharing, and ensure the security and reliability of data sharing.

[0014] Furthermore, the collected raw data undergoes preprocessing, specifically including:

[0015] During the data collection process, the collected raw data undergoes preliminary verification and cleaning, missing values ​​are identified and processed, data with format errors are corrected, and outlier data that exceeds the outlier range is removed based on preset rules.

[0016] The cleaned data is transformed according to a unified data model and standard, the test item codes from different sources are mapped to a unified standard code system, the dates, times and values ​​in different formats are unified into a standard format, and the text information is standardized.

[0017] Simultaneously, the metadata information carried by the original data is recorded; based on the preprocessed original data, a structured or semi-structured intermediate data format is formed to generate standardized laboratory data.

[0018] Furthermore, rapid data processing specifically includes:

[0019] Data writing: Standardized laboratory data is written to each storage node through the write interface of the distributed database, and the laboratory data on each storage node is split into multiple independent data shards based on the big data processing framework.

[0020] Metadata management: Logically organize laboratory data according to its dimensions and manage metadata information in a unified manner;

[0021] Parallel processing: The data processing task is divided into multiple subtasks and distributed to multiple computing nodes in the computing framework. Each computing node executes the assigned data processing task to process the data slices in parallel.

[0022] Processing result aggregation: After each computing node completes its local computation, it summarizes the parallel processed results to the master node for final aggregation, sorting, and organization to form a complete set of laboratory data.

[0023] Furthermore, the data processing task is broken down into multiple subtasks and distributed to multiple computing nodes in the computing framework, specifically including:

[0024] Use standardized laboratory data to set an upper limit on the number of sub-tasks after splitting;

[0025] The data processing task is split into multiple subtasks by using the maximum number of subtasks after splitting as a constraint.

[0026] Retrieve the CPU frequency, memory capacity, current load rate, and average time for the compute node to process the same type of task in the past for each compute node;

[0027] Normalize the CPU frequency, memory capacity, current load rate, and average time of the computing node for processing the same type of task in history for each computing node, and obtain the normalized CPU frequency, memory capacity, current load rate, and average time of the computing node for processing the same type of task in history.

[0028] The task processing capability index of each computing node is obtained by using the normalized CPU frequency, memory capacity, current load rate and the average time of the computing node to process the same type of task in the past.

[0029] Retrieve the maximum allowed processing time for each subtask;

[0030] The matching degree parameter between each subtask and each computing node is obtained by combining the maximum allowed processing time for each subtask with the task processing capacity index of each computing node.

[0031] The computing node corresponding to the maximum value of the matching degree parameter is assigned as the computing node to be allocated for each subtask.

[0032] Furthermore, using standardized laboratory data, an upper limit is set for the number of sub-tasks after decomposition, including:

[0033] Feature extraction was performed on standardized laboratory data to obtain multiple feature data.

[0034] The Shannon information entropy method is used to obtain the feature entropy corresponding to each feature data by combining the probability of each feature data appearing in the task.

[0035] Retrieve the weight value corresponding to each feature data;

[0036] The task splitting coefficient is set using the feature entropy corresponding to each feature data point of the standardized laboratory data;

[0037] Retrieve a preset number of basic subtasks; wherein the preset number of basic subtasks ranges from 10 to 100.

[0038] The upper limit of the number of subtasks after splitting is set using the task splitting coefficient and the preset number of basic subtasks.

[0039] Furthermore, outlier data exceeding a reasonable range is removed, specifically including:

[0040] Based on the equipment type of the inspection item, retrieve the corresponding parameter verification rules from the rule base and generate a data verification configuration file;

[0041] Based on the data validation configuration file, the historical data distribution characteristics of similar test items are analyzed, a dynamic validation model based on the historical data distribution characteristics is constructed, and an outlier determination threshold is generated.

[0042] Based on the data characteristics of the device type corresponding to each original data, target feature information is extracted and input into the dynamic verification model for verification. Based on the outlier determination threshold, it is determined whether there is outlier data in each original data. Data that exceeds the outlier determination threshold range is marked as outlier data.

[0043] When outlier data is detected, the distribution characteristics of the outlier data in the original data are extracted, the distribution characteristics are input into a pre-trained neural network model, matched with a historical rule amendment example library, and a rule adjustment scheme is generated by combining the current data characteristics.

[0044] Based on the generated rule adjustment scheme, calculate the compatibility coefficient between outlier data features and the original verification rules. If the compatibility coefficient is greater than the adjustment threshold, make a preset adjustment; if the compatibility coefficient is less than the adjustment threshold, modify the rules.

[0045] Based on the modified parameter verification rules, combined with the data packet size and data characteristics of the original data, the verification timestamp priority is set, and the original data is verified a second time based on the verification timestamp priority. Based on the results of the second verification, the outlier data is finally removed.

[0046] Furthermore, setting the verification timestamp priority also includes:

[0047] Detect the size of the data packet for each piece of raw data, and set a verification timestamp for the raw data based on the data packet size;

[0048] The verification timestamps are correlated with data characteristics to verify the rationality of each original data. At the same time, the trend of change of the original data and the data at adjacent time points is verified to be consistent with physiological logic.

[0049] After each piece of raw data is verified, an immutable record containing a timestamp, verification parameters, and hash value is generated.

[0050] The network transmission load of each testing device in the monitoring and testing department is dynamically adjusted based on the real-time data traffic of each testing device to optimize the priority of verification timestamps.

[0051] Furthermore, data writing also includes:

[0052] Before storing standardized laboratory data, the storage modules in the distributed database are identified, the available storage partitions and abnormal storage units are identified, and hash fingerprint verification is performed on the occupied storage units, and the occupied storage units are formatted.

[0053] By combining available storage partitions with formatted and occupied storage units, a pool of available storage resources for data storage can be obtained.

[0054] Set up several storage nodes in the available storage resource pool, select a key storage node from the several storage nodes, determine the connection relationship between the key storage node and other storage nodes, and generate a communication tree;

[0055] Data from the laboratory with the same testing items are divided into the same data shard, resulting in several data shards. Based on the storage requirements of the laboratory's business scenarios, the target storage node in the communication tree is determined.

[0056] Based on key storage nodes, the content of data transmission information is verified, a list of verification information is generated, and anomaly detection is performed on the list of verification information. When anomalies are detected, an alarm is issued.

[0057] Once no anomalies are confirmed, the standardized laboratory data in the cache is written to each target storage node in batches according to the defined sharding strategy through the write interface of the distributed database.

[0058] Furthermore, batch writing to each target storage node also includes:

[0059] During storage, data fragments are decomposed into several sub-data, the attribute information of each sub-data is determined, and the usage intensity of the sub-data is evaluated based on the attribute information;

[0060] Establish a network traffic prediction model for storage nodes, and calculate the transmission availability score of each target storage node in the future time period based on historical bandwidth usage curves and the usage intensity of sub-data.

[0061] Calculate the load balancing coefficient for each target storage node, and then calculate the weighted load balancing coefficient and compare it with the preset load balancing threshold.

[0062] When the global load balancing coefficient is determined to be less than the preset load balancing parameter, storage nodes with a load greater than the preset node processing capacity are selected as overloaded nodes, and task migration queues are built according to load pressure from high to low.

[0063] If the global load balancing coefficient is less than the preset load balancing threshold, storage units with loads lower than the preset node processing capacity are identified as light-load nodes, and resource receiving queues are built according to the resource idleness from smallest to largest.

[0064] Match the task migration queue and resource receiving queue, migrate the main content of non-time-sensitive verification data in the heavy-load node to the light-load node, and at the same time, retain the metadata information.

[0065] Furthermore, data sharing also includes controlling access to and use of data based on user identity and permissions, while encrypting shared data.

[0066] Compared with the prior art, the beneficial effects of the present invention are:

[0067] By employing a big data storage architecture and parallel processing technology, data is sharded, stored, cleaned, and integrated in parallel. This enables efficient processing of massive amounts of data generated by the laboratory, significantly shortening data processing time and meeting the timeliness requirements of rapid clinical diagnosis. Big data analytics algorithms are used to mine data correlations, extracting potential patterns such as disease trends and treatment effects from laboratory data, providing quantitative support for clinical diagnosis. Visual presentation methods improve the efficiency of interpreting test results, facilitating doctors to quickly obtain key information. A secure sharing platform is built, enabling rapid and convenient sharing of laboratory data within authorized scope, meeting the clinical need for rapid result acquisition, supporting cross-institutional collaboration, meeting medical data compliance requirements, and, based on big data technology, easily expandable to adapt to future data volume growth and increased processing needs. Attached Figure Description

[0068] Figure 1 This is a flowchart of the big data-based rapid data processing and sharing method for laboratory departments according to the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] To address the technical challenges of existing centralized data processing architectures when dealing with the terabytes of data generated daily by the laboratory, including insufficient data sharding and parallel processing capabilities, inability to dynamically adjust outlier detection rules, low storage resource utilization, and a lack of unified standard protocols and automated processes for cross-institutional data sharing, please refer to [link to relevant documentation]. Figure 1 This embodiment provides the following technical solution:

[0071] Methods for rapid data processing and sharing in laboratory medicine based on big data include:

[0072] Laboratory Data Acquisition: Based on the constructed data acquisition interface, communication connections are established with various laboratory equipment. This interface is compatible with the communication protocols of various laboratory equipment, such as various automated biochemical analyzers, blood analyzers, immunoassay analyzers, LIS (Laboratory Information System), etc. It collects raw data generated by various laboratory equipment in real time during the testing process, including patient identification information, specimen identification information, test item codes, raw measurement values, quality control data, equipment status information, etc. At the same time, the collected raw data is preprocessed to form standardized laboratory data.

[0073] Rapid data processing: Based on a distributed storage database, a data sharding and redundant storage strategy is adopted to distribute standardized laboratory data across various storage nodes. The stored data is cleaned, transformed, and integrated to improve data quality, eliminate noise, errors, and inconsistencies, and provide high-quality data support for data analysis and sharing. According to the data analysis and sharing needs, format conversion and integration are performed to form a complete set of laboratory data.

[0074] Data Analysis: Utilizing big data analytics technologies and algorithms, we perform data analysis and feature extraction on laboratory data sets to uncover the potential value and patterns within the data, providing strong support for clinical diagnosis and research decision-making. This includes understanding the basic characteristics and distribution of laboratory data, predicting disease development trends and treatment effects, and discovering correlations between data. The analysis results are presented in intuitive charts, reports, and other formats for easy user understanding and use.

[0075] Data Sharing: Construct a data sharing platform and establish a data sharing link between the platform and the distributed storage database to achieve real-time data sharing; simultaneously, establish data sharing standards and processes, clarify the scope, methods, and responsibilities of data sharing, and ensure the security and reliability of data sharing; adopt access control technology to control data access and use based on user identity and permissions, ensuring that only authorized users can access and use relevant data; and employ data encryption technology to encrypt shared data to prevent it from being stolen or tampered with during transmission and storage.

[0076] In this embodiment, by employing a big data storage architecture and parallel processing technology, data is stored in shards and cleaned and integrated in parallel. This enables efficient processing of massive amounts of data generated by the laboratory, significantly shortening data processing time and meeting the timeliness requirements of rapid clinical diagnosis. Big data analysis algorithms are used to mine data correlations, extracting potential patterns such as disease trends and treatment effects from the test data, providing quantitative support for clinical diagnosis. The visualization presentation method improves the efficiency of interpreting test results, making it easier for doctors to quickly obtain key information. A secure sharing platform is built, enabling rapid and convenient sharing of laboratory data within the authorized scope, meeting the clinical need for rapid results acquisition, supporting cross-institutional collaboration, meeting medical data compliance requirements, and being built based on big data technology, it can be easily expanded to adapt to future data volume growth and increased processing needs.

[0077] In this embodiment, the collected raw data is preprocessed, specifically including:

[0078] During the data collection process, the collected raw data undergoes preliminary verification and cleaning, missing values ​​are identified and processed, data with format errors are corrected, and outlier data that exceeds the outlier range is removed based on preset rules.

[0079] The cleaned data is transformed according to a unified data model and standard. The test item codes from different sources are mapped to a unified standard code system, such as LOINC and ICD. Dates, times, and values ​​in different formats are unified into a standard format. Text information is standardized, such as removing redundant characters and unifying terminology, to ensure that the collected data is consistent and standardized.

[0080] At the same time, record metadata information such as the collection time and device identifier carried by the raw data;

[0081] Based on the preprocessed raw data, structured or semi-structured intermediate data formats are formed to generate standardized laboratory data.

[0082] In this embodiment, outlier data that exceeds a reasonable range is removed, specifically including:

[0083] Based on the equipment type of the inspection item, the corresponding parameter verification rules are retrieved from the rule base to generate a data verification configuration file, which includes key information such as equipment model, parameter upper and lower limits, and verification algorithm.

[0084] Based on the data validation configuration file, the historical data distribution characteristics of similar test items are analyzed, a dynamic validation model based on the historical data distribution characteristics is constructed, and an outlier determination threshold that is adjusted in real time according to the data distribution characteristics is generated.

[0085] Based on the data characteristics of the device type corresponding to each original data, target feature information is extracted and input into the dynamic verification model for verification. Based on the outlier determination threshold, it is determined whether there is outlier data in each original data. Data that exceeds the outlier determination threshold range is marked as outlier data.

[0086] When outlier data is detected, the distribution characteristics of the outlier data in the original data are extracted, such as peak value, dispersion, and time series trend. The distribution characteristics are then input into a pre-trained neural network model and matched with a historical rule amendment example library. The model is then combined with the current data characteristics to generate a rule adjustment scheme and output parameters including adjustment magnitude and scope of impact.

[0087] Based on the generated rule adjustment scheme, calculate the compatibility coefficient between outlier data features and the original verification rules. If the compatibility coefficient is greater than the adjustment threshold, make a preset adjustment range, such as fine-tuning the numerical range. If the compatibility coefficient is less than the adjustment threshold, make rule modifications, such as adding a "clinical diagnosis association verification" condition.

[0088] Based on the modified parameter verification rules, combined with the data packet size and data characteristics of the original data, the verification timestamp priority is set, and the original data is verified a second time based on the verification timestamp priority. Based on the results of the second verification, the outlier data is finally removed.

[0089] In this embodiment, setting the verification timestamp priority also includes:

[0090] Detect the size of the data packet for each piece of raw data, and set a verification timestamp for the raw data based on the data packet size;

[0091] The verification timestamp is correlated with data characteristics to verify the rationality of each original data. At the same time, the trend of change of the original data and the data at adjacent time points is checked to see if it conforms to physiological logic. For example, in the medical testing scenario, the changes of human physiological indicators are usually gradual. If an indicator shows an abnormal jump in a short period of time, it needs to be marked and checked.

[0092] After each piece of raw data is verified, an immutable record containing a timestamp, verification parameters, and hash value is generated.

[0093] The network transmission load of each testing device in the monitoring laboratory is dynamically adjusted based on the real-time data traffic of each device to optimize the verification timestamp priority. By employing a "sampling verification + key parameter precision verification" strategy for high-traffic devices (such as biochemical analyzers) and implementing "full-volume in-depth verification" for low-traffic devices (such as coagulation analyzers), the overall verification efficiency is improved by more than 30% while ensuring verification quality.

[0094] In this embodiment, the mean and standard deviation of the current batch of data are calculated in real time for the blood routine white blood cell count data. Combined with the fluctuation range of similar data in the previous 30 days, the outlier data judgment threshold is automatically adjusted, which improves the accuracy of abnormal data identification by more than 20%.

[0095] In this embodiment, for the creatinine detection data of the biochemical analyzer, if outliers appear three times in a row and the distribution characteristics match the "reagent expiration" pattern, the dynamic verification model automatically incorporates the "reagent expiration date verification" parameter into the verification rules, thereby achieving the autonomous evolution of the rules.

[0096] In this embodiment, if a patient's blood glucose test data fluctuates by more than 50% within 30 minutes and there is no corresponding insulin injection record, a spatiotemporal correlation verification warning is triggered, indicating that there may be a risk of sample confusion.

[0097] In this embodiment, after the antibody detection data of the immune test is verified, the data characteristics, verification rule version number, verification time and other information are generated into a Merkle tree hash value and anchored to the blockchain node to ensure that the data verification process is traceable and tamper-proof, and meets the medical data compliance requirements.

[0098] In this embodiment, data is processed rapidly, specifically including:

[0099] Data writing: Standardized laboratory data is written to each storage node through the write interface of the distributed database, and the laboratory data on each storage node is split into multiple independent data shards based on the big data processing framework.

[0100] Metadata management: Logically organize laboratory data based on dimensions such as timestamps, patient IDs, and test types, for example, by creating partitions by time series or indexes by patient IDs; and manage metadata information in a unified manner.

[0101] Parallel processing: The data processing task is split into multiple subtasks and distributed to multiple computing nodes in the Spark distributed computing framework. Each computing node executes the assigned data processing task to process the data shards in parallel.

[0102] Processing result aggregation: After each computing node completes its local computation, it summarizes the parallel processed results to the master node for final aggregation, sorting, and organization to form a complete set of laboratory data.

[0103] Specifically, the data processing task is broken down into multiple subtasks and distributed to multiple computing nodes in the computing framework, including:

[0104] Use standardized laboratory data to set an upper limit on the number of sub-tasks after splitting;

[0105] The data processing task is split into multiple subtasks by using the maximum number of subtasks after splitting as a constraint.

[0106] Retrieve the CPU frequency, memory capacity, current load rate, and average time for the compute node to process the same type of task in the past for each compute node;

[0107] Normalize the CPU frequency, memory capacity, current load rate, and average time of the computing node for processing the same type of task in history for each computing node, and obtain the normalized CPU frequency, memory capacity, current load rate, and average time of the computing node for processing the same type of task in history.

[0108] The task processing capability index of each computing node is obtained by using the normalized CPU frequency, memory capacity, current load rate and the average time of the computing node to process the same type of task in the past.

[0109] The task processing capability index corresponding to each computing node is obtained by the following formula:

[0110]

[0111] Where P represents the number of task processing capacity indicators corresponding to each computing node; f represents the normalized CPU frequency; m represents the normalized memory capacity; L represents the normalized current load rate; and t represents the normalized average time for computing nodes to process the same type of task in the past.

[0112] Retrieve the maximum allowed processing time for each subtask;

[0113] The matching degree parameter between each subtask and each computing node is obtained by combining the maximum allowed processing time for each subtask with the task processing capacity index of each computing node.

[0114] The matching degree parameter is obtained by the following formula:

[0115]

[0116] Among them, A ij represents the matching degree parameter between the j-th computing node and the i-th subtask; r represents the first adaptation weight, and has a value of 0.4; k represents the second adaptation weight, and has a value of 0.6; t represents the average time for computing nodes to process the same type of task in the past after normalization; t g P represents the maximum allowed processing time for each subtask; S represents the number of task processing capacity indicators for each computing node; and S represents the amount of data for each subtask.

[0117] The computing node corresponding to the maximum value of the matching degree parameter is assigned as the computing node to be allocated for each subtask.

[0118] The technical effects of the above solution are as follows: By rationally splitting tasks based on standardized laboratory data and setting an upper limit on the number of sub-tasks, resource waste and inefficiency caused by excessive or insufficient task splitting are avoided, laying the foundation for efficient processing. Core performance parameters of the computing nodes, such as CPU frequency, memory capacity, current load rate, and average time for processing similar tasks in history, are normalized to derive task processing capacity indicators, ensuring a more accurate assessment of node processing capabilities. The matching degree is calculated by combining the maximum allowable processing time of sub-tasks with the node task processing capacity indicators, and sub-tasks are assigned to the nodes with the highest matching degree, achieving optimal task-node adaptation, reducing task waiting and processing time, and accelerating the overall processing progress. Simultaneously, task splitting based on standardized laboratory data unifies the basic standards for data processing, reducing processing barriers caused by differences in data formats across departments and institutions, and providing adaptability support for data sharing. The scientific allocation mechanism based on the actual performance of computing nodes can fully mobilize distributed computing resources across departments and institutions, avoiding resource idleness, allowing the computing resources of various entities to collaboratively participate in laboratory data processing, thereby promoting the efficiency of data flow and sharing in the processing stage. Through efficient task processing and rational resource utilization, the cycle from data generation to availability has been shortened, enabling data to be transferred more quickly between different departments and institutions, and further improving sharing efficiency.

[0119] Specifically, using standardized laboratory data, an upper limit is set for the number of sub-tasks after decomposition, including:

[0120] Feature extraction was performed on standardized laboratory data to obtain multiple feature data.

[0121] The Shannon information entropy method is used to obtain the feature entropy corresponding to each feature data by combining the probability of each feature data appearing in the task.

[0122] Retrieve the weight value corresponding to each feature data;

[0123] The task splitting coefficient is set using the feature entropy corresponding to each feature data point of the standardized laboratory data;

[0124] The task splitting coefficient is obtained using the following formula:

[0125]

[0126] Where K represents the task splitting coefficient; n represents the number of feature data; H i α represents the feature entropy corresponding to the i-th feature data. i This represents the weight value corresponding to the i-th feature data;

[0127] Retrieve a preset number of basic subtasks; wherein the preset number of basic subtasks ranges from 10 to 100.

[0128] The upper limit of the number of subtasks after splitting is set using the task splitting coefficient and the preset number of basic subtasks;

[0129] The upper limit of the number of subtasks after splitting is obtained by the following formula:

[0130]

[0131] Where N represents the upper limit of the number of subtasks after splitting; K represents the task splitting coefficient; and M represents the preset number of basic subtasks.

[0132] The technical effects of the above solution are as follows: Based on feature extraction from standardized laboratory data and Shannon information entropy calculation of feature entropy, task splitting coefficients are determined by combining feature weights. Furthermore, an upper limit on the number of subtasks is set based on a preset basic number of subtasks. This ensures that the number of subtasks is compatible with the feature complexity and importance of the laboratory data, avoiding increased node scheduling costs due to an excessive number of subtasks or insufficient resource utilization due to an insufficient number. This provides a reasonable basis for efficient task allocation and processing, indirectly improving overall processing speed. Simultaneously, using standardized laboratory data for feature extraction and task splitting calculations unifies the baseline for early data processing, reducing task splitting adaptation problems caused by differences in laboratory data formats or feature definitions across departments and institutions. This provides standardized support for data processing connections between different entities. Furthermore, effectively improving the rationality of the upper limit setting for the number of subtasks allows the split subtasks to better adapt to the scale of distributed computing resources across departments and institutions, facilitating collaborative task undertaking by various entities' computing resources. This reduces data processing bottlenecks caused by mismatches between task scale and resources, thereby improving the flow of laboratory data in the processing stage and further enhancing data sharing efficiency.

[0133] In this embodiment, data writing also includes:

[0134] Before storing standardized laboratory data, the storage modules in the distributed database are identified, the available storage partitions and abnormal storage units are identified, and hash fingerprint verification is performed on the occupied storage units, and the occupied storage units are formatted.

[0135] By combining available storage partitions with formatted and occupied storage units, a pool of available storage resources for data storage can be obtained.

[0136] Set up several storage nodes in the available storage resource pool, select a key storage node from the several storage nodes, determine the connection relationship between the key storage node and other storage nodes, and generate a communication tree;

[0137] Data from the laboratory with the same test items are divided into the same shard, resulting in several data shards. Based on the storage needs of the laboratory's business scenarios, the target storage nodes in the communication tree are determined. For example, time-sensitive emergency test data is stored in high read / write performance nodes close to key storage nodes, while historical test data related to scientific research is stored in nodes with large capacity and low cost.

[0138] Based on key storage nodes, the content of data transmission information is verified, a list of verification information is generated, and anomaly detection is performed on the list of verification information. When anomalies are detected, an alarm is issued.

[0139] Once no anomalies are confirmed, the standardized laboratory data in the cache is written to each target storage node in batches according to the defined sharding strategy through the write interface of the distributed database.

[0140] In this embodiment, batch writing to each target storage node further includes:

[0141] During storage, data shards are decomposed into several sub-data, such as sub-data based on test items, patient ID, or timestamps; the attribute information of each sub-data is determined, including the test item type (complete blood count, biochemical test, etc.), data generation time, data urgency (emergency data, routine data), etc. The usage intensity of the sub-data is evaluated based on the attribute information, such as emergency data, which has a high usage intensity due to the need for rapid access;

[0142] Establish a network traffic prediction model for storage nodes. Based on historical bandwidth usage curves and the usage intensity of sub-data, calculate the transmission availability score of each target storage node for future periods. For example, if a storage node experiences a high bandwidth occupancy rate of up to 80% between 10:00 AM and 12:00 PM daily due to a large number of data queries, prioritize matching sub-data with high usage intensity to storage nodes with sufficient bandwidth and fast response speed.

[0143] Calculate the load balancing coefficient for each target storage node, and then calculate the weighted load balancing coefficient and compare it with the preset load balancing threshold.

[0144] When the global load balancing coefficient is determined to be less than the preset load balancing parameter, storage nodes with a load greater than the preset node processing capacity are selected as overloaded nodes, and task migration queues are built according to load pressure from high to low.

[0145] If the global load balancing coefficient is less than the preset load balancing threshold, storage units with loads lower than the preset node processing capacity are identified as light-load nodes, and resource receiving queues are built according to the resource idleness from smallest to largest.

[0146] The task migration queue and resource receiving queue are matched to migrate the main content of non-time-sensitive verification data in the heavy-load node to the light-load node. At the same time, metadata information such as data index and access control policies are preserved.

[0147] In this embodiment, standardized laboratory data is sharded and stored through a distributed database write interface. Data shards are further split using a big data processing framework, and the Spark distributed computing framework is used to break down processing tasks into sub-tasks for parallel execution. Finally, the processing results are aggregated to form a complete dataset, effectively improving the processing efficiency and reliability of massive amounts of laboratory data. During batch data writing, the accuracy and security of data storage are ensured. Furthermore, by decomposing data shards and evaluating the usage intensity of sub-data, calculating load balancing coefficients, and dynamically migrating non-time-sensitive data from heavily loaded nodes to lightly loaded nodes, intelligent scheduling and load balancing of storage resources are achieved. This optimizes the resource utilization of storage nodes, shortens the read / write response time for time-sensitive data such as emergency data, and improves overall data processing efficiency and storage reliability.

[0148] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for rapid processing and sharing of laboratory data based on big data, characterized in that: include: Laboratory data acquisition: Based on the constructed data acquisition interface, communication connections are established with various testing equipment to collect raw data generated by each testing equipment in the laboratory during the testing process in real time. At the same time, the collected raw data is preprocessed to form standardized laboratory data. Rapid data processing: Based on a distributed storage database, data sharding and redundant storage strategies are adopted to distribute standardized laboratory data across various storage nodes. According to data analysis and sharing needs, format conversion and integration are performed to form a complete laboratory data set. Among them, the parallel splitting and allocation of data processing tasks are realized through a distributed computing framework, and the data storage and processing resource scheduling is dynamically optimized in combination with the performance of storage nodes. Data analysis: Perform data analysis and feature extraction on the laboratory data set to uncover the potential value and patterns in the data, and present the analysis results in an intuitive form; Data sharing: Build a data sharing platform, establish a data sharing link between the data sharing platform and the distributed storage database to achieve real-time data sharing; at the same time, establish data sharing standards and processes, and clarify the scope, methods and responsibilities of data sharing. The rapid data processing includes: data writing; metadata management; parallel processing; and processing result aggregation. The data processing task is broken down into multiple subtasks and distributed to multiple computing nodes in the computing framework, specifically including: Use standardized laboratory data to set an upper limit on the number of sub-tasks after splitting; The data processing task is split into multiple subtasks by using the maximum number of subtasks after splitting as a constraint. Retrieve the CPU frequency, memory capacity, current load rate, and average time for the compute node to process the same type of task in the past for each compute node; Normalization is performed on each data point; the normalized data is then used to obtain the task processing capability index for each computing node. The formula is as follows: Where P represents the number of task processing capacity indicators corresponding to each computing node; f represents the normalized CPU frequency; m represents the normalized memory capacity; L represents the normalized current load rate; and t represents the normalized average time for computing nodes to process the same type of task in the past. Retrieve the maximum allowed processing time for each subtask and obtain the matching degree parameter between each subtask and each computing node; This matching degree parameter is obtained using the following formula: Among them, A ij represents the matching degree parameter between the j-th computing node and the i-th subtask; r represents the first adaptation weight, and has a value of 0.4; k represents the second adaptation weight, and has a value of 0.6; t represents the average time for computing nodes to process the same type of task in the past after normalization; t g P represents the maximum allowed processing time for each subtask; S represents the number of task processing capacity indicators for each computing node; and S represents the amount of data for each subtask. The computing node corresponding to the maximum value of the matching degree parameter is assigned as the computing node to be allocated for each subtask.

2. The method for rapid processing and sharing of laboratory data based on big data as described in claim 1, characterized in that, The collected raw data undergoes preprocessing, specifically including: During the data collection process, the collected raw data undergoes preliminary verification and cleaning, missing values ​​are identified and processed, data with format errors are corrected, and outlier data that exceeds the outlier range is removed based on preset rules. The cleaned data is transformed according to a unified data model and standard, the test item codes from different sources are mapped to a unified standard code system, the dates, times and values ​​in different formats are unified into a standard format, and the text information is standardized. Simultaneously, the metadata information carried by the original data is recorded, and a structured or semi-structured intermediate data format is formed based on the preprocessed original data to generate standardized laboratory data.

3. The method for rapid processing and sharing of laboratory data based on big data as described in claim 1, characterized in that, The rapid data processing specifically includes: Data writing: Standardized laboratory data is written to each storage node through the write interface of the distributed database, and the laboratory data on each storage node is split into multiple independent data shards based on the big data processing framework. Metadata management: Logically organize laboratory data according to its dimensions and manage metadata information in a unified manner; Parallel processing: The data processing task is divided into multiple subtasks and distributed to multiple computing nodes in the computing framework. Each computing node executes the assigned data processing task to process the data slices in parallel. Processing result aggregation: After each computing node completes its local computation, it summarizes the parallel processed results to the master node for final aggregation, sorting, and organization to form a complete set of laboratory data.

4. The method for rapid processing and sharing of laboratory data based on big data as described in claim 1, characterized in that, Using standardized laboratory data, set an upper limit on the number of subtasks after splitting, including: Feature extraction was performed on standardized laboratory data to obtain multiple feature data. The Shannon information entropy method is used to obtain the feature entropy corresponding to each feature data by combining the probability of each feature data appearing in the task. Retrieve the weight value corresponding to each feature data; The task splitting coefficient is set using the feature entropy corresponding to each feature data point of the standardized laboratory data; The task splitting coefficient is obtained using the following formula: Where K represents the task splitting coefficient; n represents the number of feature data; H i α represents the feature entropy corresponding to the i-th feature data. i This represents the weight value corresponding to the i-th feature data; Retrieve a preset number of basic subtasks; wherein the preset number of basic subtasks ranges from 10 to 100. The upper limit of the number of subtasks after splitting is set using the task splitting coefficient and the preset number of basic subtasks; The upper limit of the number of subtasks after splitting is obtained by the following formula: Where N represents the upper limit of the number of subtasks after splitting; K represents the task splitting coefficient; and M represents the preset number of basic subtasks.

5. The method for rapid processing and sharing of laboratory data based on big data as described in claim 2, characterized in that, Outlier data that exceeds a reasonable range will be removed, specifically including: Based on the equipment type of the inspection item, retrieve the corresponding parameter verification rules from the rule base and generate a data verification configuration file; Based on the data validation configuration file, the historical data distribution characteristics of similar test items are analyzed, a dynamic validation model based on the historical data distribution characteristics is constructed, and an outlier determination threshold is generated. Based on the data characteristics of the device type corresponding to each original data, target feature information is extracted and input into the dynamic verification model for verification. Based on the outlier determination threshold, it is determined whether there is outlier data in each original data. Data that exceeds the outlier determination threshold range is marked as outlier data. When outlier data is detected, the distribution characteristics of the outlier data in the original data are extracted, the distribution characteristics are input into a pre-trained neural network model, matched with a historical rule amendment example library, and a rule adjustment scheme is generated by combining the current data characteristics. Based on the generated rule adjustment scheme, calculate the compatibility coefficient between outlier data features and the original verification rules. If the compatibility coefficient is greater than the adjustment threshold, make a preset adjustment; if the compatibility coefficient is less than the adjustment threshold, modify the rules. Based on the modified parameter verification rules, combined with the data packet size and data characteristics of the original data, the verification timestamp priority is set, and the original data is verified a second time based on the verification timestamp priority. Based on the results of the second verification, the outlier data is finally removed.

6. The method for rapid processing and sharing of laboratory data based on big data as described in claim 5, characterized in that, Setting the verification timestamp priority also includes: Detect the size of the data packet for each piece of raw data, and set a verification timestamp for the raw data based on the data packet size; The verification timestamps are correlated with data characteristics to verify the rationality of each original data. At the same time, the trend of change of the original data and the data at adjacent time points is verified to be consistent with physiological logic. After each piece of raw data is verified, an immutable record containing a timestamp, verification parameters, and hash value is generated. The network transmission load of each testing device in the monitoring and testing department is dynamically adjusted based on the real-time data traffic of each testing device to optimize the priority of verification timestamps.

7. The method for rapid processing and sharing of laboratory data based on big data as described in claim 6, characterized in that, Data writing also includes: Before storing standardized laboratory data, the storage modules in the distributed database are identified, the available storage partitions and abnormal storage units are identified, and hash fingerprint verification is performed on the occupied storage units, and the occupied storage units are formatted. By combining available storage partitions with formatted and occupied storage units, a pool of available storage resources for data storage can be obtained. Set up several storage nodes in the available storage resource pool, select a key storage node from the several storage nodes, determine the connection relationship between the key storage node and other storage nodes, and generate a communication tree; Data from the laboratory with the same testing items are divided into the same data shard, resulting in several data shards. Based on the storage requirements of the laboratory's business scenarios, the target storage node in the communication tree is determined. Based on key storage nodes, the content of data transmission information is verified, a list of verification information is generated, and anomaly detection is performed on the list of verification information. When anomalies are detected, an alarm is issued. Once no anomalies are confirmed, the standardized laboratory data in the cache is written to each target storage node in batches according to the defined sharding strategy through the write interface of the distributed database.

8. The method for rapid processing and sharing of laboratory data based on big data as described in claim 7, characterized in that, Batch writing to various target storage nodes also includes: During storage, data fragments are decomposed into several sub-data, the attribute information of each sub-data is determined, and the usage intensity of the sub-data is evaluated based on the attribute information; Establish a network traffic prediction model for storage nodes, and calculate the transmission availability score of each target storage node in the future time period based on historical bandwidth usage curves and the usage intensity of sub-data. Calculate the load balancing coefficient for each target storage node, and then calculate the weighted load balancing coefficient and compare it with the preset load balancing threshold. When the global load balancing coefficient is determined to be less than the preset load balancing parameter, storage nodes with a load greater than the preset node processing capacity are selected as overloaded nodes, and task migration queues are built according to load pressure from high to low. If the global load balancing coefficient is less than the preset load balancing threshold, storage units with loads lower than the preset node processing capacity are identified as light-load nodes, and resource receiving queues are built according to the resource idleness from smallest to largest. Match the task migration queue and resource receiving queue, migrate the main content of non-time-sensitive verification data in the heavy-load node to the light-load node, and at the same time, retain the metadata information.

9. The method for rapid processing and sharing of laboratory data based on big data as described in claim 8, characterized in that, Data sharing also includes controlling access to and use of data based on user identity and permissions, and encrypting shared data.

Images