Medical health data sharing method and service system based on high concurrency

Abstract

The invention relates to the technical field of data sharing, and discloses a medical health data sharing method and service system based on high concurrency. Comprising a data processing device, a load layer and a server cluster, the data processing device comprises a data uploading interface, a front-end processor, a data uploading module, a service library and a cache library, and the data uploading interface is used for receiving original data from different channels and uploading the original data of each channel to the front-end processor; the data uploading module is used for uploading the data in the front-end processor to a service library in a distributed manner; the business library is used for normalizing the data to obtain effective data; the cache library is used for storing valid data; the load layer is used for sending the received data access request to the server cluster; and the server cluster is used for acquiring the target data from the cache library according to the data access request, sending the target data to the load layer, and sending the target data to the client through the load layer. According to the invention, the high-concurrency data sharing service of the data is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of medical health data sharing, and in particular to a medical health data sharing method and service system based on high concurrency. Background Art

[0002] After years of informatization construction, the health industry has formed a massive amount of business, process and management data, especially realizing the interconnection between industry authorities and medical institutions, laying the foundation for business modeling supported by industry applications. At the same time, in the construction goals of the National Health Informatization Project, by carrying out regional-level sharing services for national public health data, such as electronic medical records inspection and testing data, the interconnection and sharing value of data can be improved.

[0003] The public health data sharing for the whole society has a large user base. Through the demand analysis of shared interface usage, it is found that the demand for each interface call volume varies greatly. Some have large-scale access, while others have high concurrency, large traffic and strong continuity, with daily call volume exceeding hundreds of millions of times. When the general interface service is subjected to an ultra-high stress test of 10,000 times per second, the error rate is difficult to control below 1%. Moreover, the shared interface service faces updates and upgrades under high concurrency pressure, and it needs to be imperceptible to users to ensure business continuity and stability. Summary of the invention

[0004] In view of this, the present invention provides a medical health data sharing method and service system based on high concurrency to solve the problem of high concurrency pressure faced by the national public medical health data sharing service in the related technology.

[0005] In a first aspect, the present invention provides a medical health data sharing service system based on high concurrency, the system comprising: a data processing device, a load layer and a server cluster, wherein the data processing device comprises: a data upload interface, a front-end processor, a data upload module, a business library, and a cache library;

[0006] The data upload interface is used to receive raw data from different channels, and upload the raw data of each channel to the front-end respectively; for raw data with a data volume greater than a preset value, the data upload interface compresses the raw data, and uploads the compressed data to the front-end; the data upload module is used to upload the data in the front-end to the business library in a distributed manner, and the business library is used to normalize the data to obtain valid data; the cache library is used to store the valid data;

[0007] The load layer is used to receive data access requests from clients and send the data access requests to the server cluster;

[0008] The server cluster is used to obtain the target data from the cache according to the data access request, send the target data to the load layer, and send the target data to the client through the load layer. The medical health data sharing service system based on high concurrency provided by the present invention can upload the original data of different channels to the front-end machine through the data upload interface, and then use the data upload module to upload the data of different formats in the front-end machine to the business library in a distributed and high-concurrency manner. The business library screens the data for validity, clears invalid data, reduces the system data volume pressure, and stores valid data in the cache. The cache can improve the response speed of data access. The load layer provides an access address to accept high-concurrency access requirements, and the server cluster can respond to high-concurrency access requirements. The system can realize high-concurrency data writing into the business library, clear unnecessary invalid data, and meet high-concurrency access requirements.

[0009] In an optional embodiment, the cache library includes: a first-level cache library and a second-level cache library, wherein the first-level cache library is used to store valid data; the second-level cache library is used to store hot data, which is valid data with an access frequency greater than a preset frequency; the access level of the second-level cache library is higher than that of the first-level cache library.

[0010] The high-concurrency medical health data sharing service system provided by the present invention provides an efficient data cache library, wherein the secondary cache library has priority when reading data, and therefore, data with relatively high access requirements are stored in the secondary cache library, and all other medical data are stored in the primary cache library. The format of the secondary cache library is conducive to high-concurrency access to data, and its performance is higher than that of a single level.

[0011] In an optional implementation, the load layer includes: soft load and hard load, wherein one hard load corresponds to multiple soft loads, the hard load faces the client; the soft load faces the server cluster.

[0012] The medical health data sharing service system based on high concurrency provided by the present invention adopts a combination of soft load and hard load in the load layer, in which one hard load corresponds to multiple soft loads. Multiple soft loads can realize one soft load proxy for one shared server cluster as much as possible to ensure isolation between services and avoid mutual interference; one hard load can improve the access performance of a single low-to-noise channel and ensure the uniqueness of the client access address.

[0013] In an optional embodiment, the data upload module includes a main node and multiple child nodes, wherein the main node is used to split the data in the front-end machine according to the data volume, and form multiple tasks according to the split results, and send each task to each child node respectively; the child node is used to execute the task and write the data corresponding to the task into the business library.

[0014] The medical and health data sharing service system based on high concurrency provided by the present invention. The data upload module includes a master node and multiple slave nodes. The master node distributes, and multiple slave nodes jointly write the data to be uploaded into the service database. This way of writing into the service database is a distributed way. Compared with the original way of directly writing data into the service database by one node, using the distributed way to write data into the database reduces the pressure on one node to write data into the database; no matter how much data there is, as long as more nodes are used, the data can be written into the service database in the fastest time, realizing high-concurrency writing of data.

[0015] In an alternative embodiment, it includes: if the data type of the data is structured data, the data upload interface uploads the data to the relational database deployed on the front-end machine.

[0016] In an alternative embodiment, it includes: if the data type of the data is unstructured data, the data is uploaded to the file server.

[0017] In an alternative embodiment, the service database is also used to determine the collection time of each data, and delete the data that exceeds the preset time limit according to the collection time of each data.

[0018] The medical and health data sharing service system based on high concurrency provided by the present invention. The service database will regularly confirm the collection time of the internal data, and then measure the collection time of the data according to the preset time limit. If the time from the collection time to the current time has exceeded the preset time limit, it proves that the data has passed the validity period. Then the service database will promptly select to clean up these expired data, reduce the data accumulation in the service database, relieve the system pressure, and ensure the availability of the data in the service database.

[0019] In an alternative embodiment, each channel is set with a preset daily settlement capacity. If the data volume provided by a channel is greater than the preset daily settlement capacity, the data upload interface fuses the channel. After the channel is fused, the channel will no longer transport data on the same day.

[0020] The medical and health data sharing service system based on high concurrency provided by the present invention. All channels used for transmitting data are set with a preset daily settlement capacity. When the data of a certain channel exceeds the set preset daily settlement capacity value, it proves that the data transmitted by this channel has an abnormality and exceeds the reasonable data volume range. The channel is directly fused to avoid a large amount of invalid data being transmitted to the system service database, relieve the pressure on the service database, and ensure the rationality of the service data in the database.

[0021] In a second aspect, the present invention provides a method for sharing medical and health data based on high concurrency. The method includes:

[0022] Receiving raw data from different channels;

[0023] Compare the data volume corresponding to the data of each channel with a preset value, and compress the original data with a data volume greater than the preset value to obtain compressed data;

[0024] Upload the original data with a data volume less than or equal to the preset value and the compressed data to the service library in a distributed manner, so that the service library normalizes the original data with a data volume less than or equal to the preset value and the compressed data to obtain valid data, and stores the valid data in the cache library.

[0025] In a third aspect, the present invention provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the methods performed by the components in the high-concurrency-based medical and health data sharing service system according to the first aspect or any corresponding embodiment thereof.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored. The computer instructions are used to cause a computer to execute the methods performed by the components in the high-concurrency-based medical and health data sharing service system according to the first aspect or any corresponding embodiment thereof. Description of the Drawings

[0027] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0028] Figure 1 It is a schematic structural diagram of a high-concurrency-based medical and health data sharing service system according to an embodiment of the present invention;

[0029] Figure 2 It is a schematic diagram of the high-concurrency access process executed by a high-concurrency-based medical and health data sharing service system according to an embodiment of the present invention;

[0030] Figure 3 It is a flowchart of a high-concurrency-based medical and health data sharing method according to an embodiment of the present invention;

[0031] Figure 4 It is a schematic hardware structure diagram of a computer device according to an embodiment of the present invention. Detailed Embodiments

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present invention.

[0033] According to an embodiment of the present invention, an embodiment of a method and service system for sharing medical and health data based on high concurrency is provided. It should be noted that the technical features involved in different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] Since the sharing of national public medical and health data is for the whole society and the base number of users is large. Through the analysis of the usage requirements of sharing interfaces, it is found that the demand differences in the call volume of each interface province are relatively large. Some have a large number of accesses, some have high concurrency, large traffic, and strong persistence. Also, since the error rate of general interface services is difficult to control below 1% when undergoing ultra-high pressure tests, therefore, the present invention provides a medical and health data sharing service system based on high concurrency to solve the problem of high concurrency pressure and meet the requirements of high-concurrency transmission, access, and reading of data.

[0035] A medical and health data sharing service system based on high concurrency provided in this embodiment can be applied to the sharing service of national public medical and health data. Figure 1 It is a structural diagram of a medical and health data sharing service system based on an embodiment of the present invention, as Figure 1 shown:

[0036] A data processing device, a load layer, and a server cluster, wherein the data processing device includes: a data upload interface 1, a front-end machine 2, a data upload module 3, a service database 4, a cache database 5, a server cluster 6, and a load layer 7.

[0037] The data upload interface 1 is used to receive raw data from different channels and upload the raw data of each channel to the front-end machine 2 respectively; for raw data with a data volume greater than a preset value, the data upload interface 1 compresses the raw data and uploads the compressed data obtained by compression to the front-end machine 2.

[0038] In an optional embodiment, the data received by the system provided in this embodiment comes from different regions, and the data upload methods and contents can be different. For example, the data can come from different provinces and different departments, and the uploaded data can be in the form of text or other forms such as pictures.

[0039] In an optional embodiment, the secure and effective isolation of the network environment is achieved by setting up a front-end machine 2 or a file server. Due to the wide range and diverse types of data sources, different upload modes can be adopted. If it is structured data, it is uploaded to the relational database (such as MYSQL) deployed on the front-end machine 2; if it is unstructured data, the data is uploaded to the file server. The data upload module 3 is used to upload the data in the front-end machine 2 to the business database 4 in a distributed manner.

[0040] In an optional embodiment, according to the data storage mode, a medical and health data exchange service is developed accordingly. For the data uploaded to the relational database, the ETL tool is used for extraction; for the data files uploaded to the file server, a self-developed specific parsing program is used for parsing and extraction.

[0041] In an optional embodiment, the data upload module 2 uploads data in a distributed manner (i.e., configuring scheduling tasks): the service is deployed on multiple servers, such as a main node service and multiple sub-node services, and the number of sub-nodes is determined according to indicators such as the scale of data exchange in actual use. Among them, the main node service is responsible for the configuration, generation, distribution, and monitoring of tasks (this task is understood as data upload). First, on the tool, the data exchange task is configured through a visual Web operation interface and stored in the database. Secondly, after the task scheduling starts and before the data processing begins, the task sharding function code program of the main node will use SQL to count and sort out the amount of data to be processed in this task, and reasonably split the data according to the amount of processed data (such as average distribution, etc.) into specific task data segments, and each task segment data is non-repetitive. Finally, using multi-threading technology, the split tasks are distributed to different sub-node services for concurrent execution. After the sub-node service finishes execution, the processing results are written into the database. This distributed data processing method greatly improves the data processing efficiency.

[0042] The business database 4 is used to normalize the data to obtain valid data.

[0043] In an alternative embodiment, the scale of medical and health data is huge, and it comes from multiple sources and has multiple types. Therefore, it is necessary to perform normalization processing first. Normalization processing is to establish a standardized model specification according to information such as the definition, format, attribute range, and association relationship with other data of the data, and then establish it in tabular form to form the business database 4. In an alternative embodiment, the built business database 4 contains a large amount of medical and health data, and not all of these data are necessarily available and valid data. In the event of a large-scale health emergency, the data volume in the business database 4 is even more uncontrollable. Therefore, it is necessary to clean the data, remove redundant, invalid, and non-conforming data, and only retain the valid data. By setting a unified data increment threshold through the system interface, or automatically calculating the increment threshold by the system according to the frequency of the most recently uploaded data, if the threshold is exceeded, an active alarm will be issued. On the one hand, it can be actively fused, and on the other hand, it can remind the operation and maintenance personnel to manually fuse to achieve control of the impact of emergencies. At the same time, the interface can configure the validity period of the data, the data volume size threshold, the data exception rules, actively restrict data publishing and clean up the published historical data, retain the valid data, and comprehensively ensure the continuity and stability of the medical and health data sharing service.

[0044] The cache database 5 is used to store valid data.

[0045] The load layer 7 is used to receive the data access requests from the client and send the data access requests to the server cluster 6.

[0046] In an alternative embodiment, the load layer is used to receive the access requirements of the access party. The load layer provides an address interface externally. The access party sends the access requirements to the load layer at this address, and the load layer then sends the access requirements to the server cluster.

[0047] The server cluster 6 is used to obtain the target data from the cache database 5 according to the data access request, send the target data to the load layer 7, and send the target data to the client through the load layer 7.

[0048] In an alternative embodiment, the server cluster is to connect multiple servers through a fast communication link. From the outside, these servers seem to be working like a single server. Internally, the incoming load is dynamically distributed to these node machines through a certain mechanism, so as to achieve the high performance of a super server.

[0049] In an alternative embodiment, the medical and health data access service is developed using a popular Web framework, deployed in a virtual machine environment, and supports Docker containerization deployment. It adopts a mode of single-service independent deployment on multiple virtual servers. This mode is flexible in deployment and easy to expand. It can determine the scale of the server cluster according to the scale of data access volume.

[0050] The high-concurrency-based medical and health data sharing service system provided by this embodiment can upload the original data from different channels to the front-end machine 2 through the data upload interface 1, and then use the data upload module 3 to distributively and highly concurrently upload the data in different formats in the front-end machine 2 to the business database 4. The business database 4 performs validity screening on the data, clears the invalid data, reduces the pressure of the system data volume, and stores the valid data in the cache database 5. The cache database 5 is a secondary high-efficiency cache database. By storing the hot data in the secondary cache database, it can be preferentially read during access, improving the response speed during data access. The load layer provides an access address through the hard load. The hard load has strong processing capabilities and can receive high-concurrency access requirements. The server cluster 6 can then respond to the high-concurrency access requirements. This system can achieve high-concurrency data writing to the business database 4, clear the unnecessary invalid data, and meet the high-concurrency access requirements.

[0051] In some alternative embodiments, the cache database 5 includes: a primary cache database and a secondary cache database. Among them, the primary cache database is used to store valid data; the secondary cache database is used to store hot data, and the hot data is valid data whose access frequency is greater than the preset frequency; the access level of the secondary cache database is higher than that of the primary cache database.

[0052] Specifically, based on requirements such as data security, persistence, and backup, medical and health business data exchanges are usually stored in a conventional database. However, a conventional database cannot effectively support high-concurrency data queries, and a high-performance data cache database needs to be built. Due to the memory storage limit of a single server, it is not convenient for later expansion. The data is stored in multiple REDIS cluster services with small single-node memories. For data with high access volume, it is stored in the MemoryCache cache, and the memory size is controlled.

[0053] The high-concurrency-based medical and health data sharing service system provided by this embodiment provides an efficient data cache database. Among them, since the secondary cache database has priority when reading data, the data with relatively high access requirements is stored in the secondary cache database, and all other medical data is stored in the primary cache database. The format of the secondary cache database is conducive to high-concurrency access to data, and its performance is higher than that of a single-level cache database.

[0054] In some alternative embodiments, the load layer includes: a soft load and a hard load. Among them, one hard load corresponds to multiple soft loads. The hard load faces the client; the soft load faces the server cluster 6.

[0055] Specifically, after the access party in this embodiment puts forward an access requirement, it goes through the hard load to the soft load, and then the access requirement reaches the server cluster 6, and the server cluster 6 provides data access; the hard load faces the client, and one hard load ensures that there is only one access address. Multiple soft loads correspond to different service ends. Since one hard load in the load layer corresponds to multiple soft load architectures, the load layer adopts a polling mechanism to execute access requests.

[0056] In an alternative embodiment, the polling mechanism is a commonly used load balancing strategy. The core idea is that the servers take turns to process user requests to make the number of requests processed by each server as the same as possible. For example, there are 6 requests numbered from request 1 to 6, and there are 3 servers that can process the requests numbered from server 1 to 3. If the sequential polling strategy is adopted, the requests will be processed in turn according to the order of server 1, 2, and 3.

[0057] In an alternative embodiment, the system in this embodiment builds a soft load service, centrally proxies the scattered business service interfaces, and provides a unified export. According to the types of medical and health data, different soft load services are used for proxy respectively. For example, the load proxy for the birth certificate health data sharing service, the load proxy for the health record data sharing service, etc.

[0058] In an alternative embodiment, the system in this embodiment builds a hard load service to further improve the access performance of the single-address channel and solve the problem of extremely high concurrent access that cannot be supported by a single soft load service. For example, the soft loads such as the birth certificate health data sharing service soft load and the health record data sharing service soft load are uniformly proxied by the hard load service.

[0059] The medical and health data sharing service system based on high concurrency provided by this embodiment adopts a combination of soft load and hard load in the load layer. One hard load corresponds to multiple soft loads. The use of multiple soft loads can, as much as possible, implement the business requirements executed in a shared server cluster by one soft load, ensure the isolation between services, and avoid mutual interference; one hard load can improve the access performance of the single low-to-channel and ensure the uniqueness of the client access address.

[0060] In some alternative implementation manners, the data upload module 3 includes a master node and multiple slave nodes. Among them, the master node is used to split the data in the front-end machine 2 according to the data volume, form multiple tasks according to the splitting results, and send each task to each slave node respectively; the slave node is used to execute the task and write the data corresponding to the task into the service database 4.

[0061] Specifically, the data upload module 3 uploads data in a distributed and multi-threaded concurrent processing manner. It includes a main node service and multiple sub-node services, and the number of sub-nodes can be determined according to indicators such as the scale of data exchange in actual use. Among them, the main node service is responsible for task configuration, generation, distribution, and monitoring. The main node counts and sorts out the amount of data to be processed in this task, and reasonably splits the data according to the amount of processed data (such as average distribution, etc.) into specific task data segments, and each task segment data does not repeat each other. Finally, using multi-threaded technology, the split tasks are distributed to different sub-services for concurrent execution. After the sub-service finishes execution, the processing results are written into the database.

[0062] In the medical and health data sharing service system based on high concurrency provided by this embodiment, the data upload module 3 includes a main node and multiple sub-nodes. The main node distributes, and multiple sub-nodes jointly write the data to be uploaded into the service database 4. This way of writing into the service database 4 is a distributed way. Compared with the original way of directly writing data into the service database 4 by one node, using the distributed way to write data into the database reduces the pressure on one node to write into the database; no matter how much data there is, as long as more nodes are adopted, the data can be written into the service database 4 in the fastest time, realizing high-concurrency writing of data and greatly improving the data processing efficiency.

[0063] In some optional implementation manners, the method for uploading data to the front-end machine includes: if the data type of the data is structured data, the data upload interface 1 uploads the data to the relational database deployed on the front-end machine 2.

[0064] In an optional embodiment, if the uploaded data is structured data, that is, if the uploaded data exists in the form of types such as tables and numbers in a database table and can be represented in a two-dimensional table structure, then the data can be uploaded to the relational database deployed on the front-end machine 2.

[0065] In some optional implementation manners, if the data type of the data is unstructured data, the data is uploaded to the file server.

[0066] In an optional embodiment, if the uploaded data is unstructured data, that is, if the uploaded data exists in the form of types such as pictures and documents, then the data can be uploaded to the file server deployed on the front-end machine 2 and stored in a directory structure.

[0067] In an optional embodiment, when the data is large, a data file in a specified format (such as CSV) is used for uploading.

[0068] In some optional implementation manners, the service database 4 is also used to determine the collection time of each data, and delete the data that exceeds the preset time limit according to the collection time of each data.

[0069] Specifically, the data publishing service actively controls the timeliness of the data to be published, filters out the invalid data whose detection time is greater than the preset timeliness, and eliminates the memory occupation of the invalid data.

[0070] In the high-concurrency-based medical and health data sharing service system provided in this embodiment, the business database 4 will regularly confirm the collection time of the internal data, and then measure the data collection time according to the preset timeliness. If the time from the collection time to the current time has exceeded the preset timeliness, it proves that the data has passed the validity period. Then the business database 4 will promptly select to clean up these expired data, reduce the data accumulation volume of the business database 4, relieve the system pressure, and ensure the availability of the data in the business database 4.

[0071] In some alternative embodiments, each channel is set with a preset daily settlement capacity. If the data volume provided by a channel is greater than the preset daily settlement capacity, the data upload interface 1 fuses the channel. After the channel is fused, the channel will no longer transmit data on the same day.

[0072] Specifically, the data fusing mechanism is based on the data exchange volume of the business system within a unit time, sets the preset daily settlement capacity, and establishes a fusing mechanism. When the data exchange volume within a unit time reaches the preset daily settlement capacity, the fusing warning mechanism is triggered, and the medical and health data publishing service is notified to stop publishing the business system data during this time period.

[0073] In an alternative embodiment, data fusing includes manual fusing and automatic fusing. Automatic fusing means that the system automatically notifies the publishing service to stop publishing the business system data according to the preset daily settlement capacity. Manual fusing means that the operation and maintenance personnel regularly check the fusing warning page. When it is found that the business system data exceeds the threshold, an exception is reported, and after comprehensive analysis, the fusing is manually triggered.

[0074] In the high-concurrency-based medical and health data sharing service system provided in this embodiment, the channels for transmitting data are all set with a preset daily settlement capacity. When the data of a certain channel exceeds the set preset daily settlement capacity value, it proves that the data transmitted by this channel has an abnormality and exceeds the reasonable data volume range. The channel is directly fused, avoiding a large amount of invalid data from being transmitted to the system business database 4, relieving the pressure on the business database 4, and ensuring the rationality of the database business data. In an alternative embodiment, such as Figure 2As shown in the figure, in this embodiment, the high-concurrency-based medical and health data sharing service system divides the process of uploading data to data access into four processes, namely, the construction of medical and health data exchange, the construction of medical and health data publishing, the construction of medical and health data access, and online verification and promotion. Among them, the construction of medical and health data exchange refers to setting up a front-end machine or a file server, developing an exchange service (data upload), and constructing a business database, which have been specifically described in the above embodiments; the construction of medical and health data publishing refers to developing a publishing service, fusing, limiting traffic, developing a cleaning service, and setting up a data cache library, which have been specifically described in the above embodiments; the construction of medical and health data access is to set up a server cluster, soft load and hard load, which have been specifically described in the above embodiments; online verification and promotion means that after the high-concurrency-based medical and health data sharing service system is put into application online, it will also be verified by the data access party and continuously optimized, and executed according to the processes of service docking, service use, effect evaluation, and promotion to other users. That is, the user can submit a corresponding application to upload data to this system to achieve docking, or can also access the data; after docking, the user tries out the data access service, optimizes it according to the feedback opinions put forward by the user, as well as the feedback and suggestions on the interaction and convenience of the service; after a certain period of trial use, according to the feedback and suggestions of the user, as well as the monitoring and statistical results of the service itself, comprehensively evaluate the service use effect, laying a foundation for further application and promotion.

[0075] Figure 3 is a flowchart of a high-concurrency-based medical and health data sharing method according to an embodiment of the present invention. As Figure 3 shown, the process includes the following steps:

[0076] Step S301, receiving raw data from different channels.

[0077] In the embodiment of the present application, the data upload interface 1 in the data processing device is used to receive raw data from different channels. The raw data from these different channels has a wide range of sources, such as from different regions (different provinces, different departments, etc.), and its data forms are diverse, which can be data in other forms such as text and pictures. As the receiving end, the data upload interface 1 will perform corresponding receiving operations on the raw data transmitted from each channel, incorporate it into the data processing process of the entire system, and lay a foundation for subsequent data processing.

[0078] Step S302, comparing the data volume corresponding to the data of each channel with a preset value, and compressing the raw data with a data volume greater than the preset value to obtain compressed data.

[0079] In the embodiment of the present application, after the data upload interface 1 receives the original data of each channel, it will compare the data volume corresponding to each channel with a preset value one by one. For the original data whose data volume is greater than the preset value, the data upload interface 1 will use the corresponding compression algorithm to compress it to obtain compressed data. For example, if the preset value is set to 10MB, when the original data received from a certain channel reaches 15MB in data volume, the data upload interface 1 will start the compression program and compress the 15MB of original data according to the established compression rules (such as common lossless compression or lossy compression algorithms, specifically depending on the requirements for the data and subsequent application scenarios), and finally obtain compressed data with a smaller volume for more efficient subsequent transmission and storage operations.

[0080] Step S303: Upload the original data with a data volume less than or equal to the preset value and the compressed data to the service library in a distributed manner, so that the service library normalizes the original data with a data volume less than or equal to the preset value and the compressed data to obtain valid data, and stores the valid data in the cache library.

[0081] In the embodiment of the present application, first, the data upload module 3 will obtain the original data with a data volume less than or equal to the preset value and the compressed data obtained through step S302. Then, a distributed method (i.e., configuring a scheduling task) is used for the upload operation. In the entire server cluster 6, the service is deployed on multiple servers, including a main node service and multiple sub-node services. The number of sub-nodes will be determined according to indicators such as the scale of data exchange in actual use.

[0082] The main node service is responsible for the configuration, generation, distribution, and monitoring of tasks (here the task refers to the data upload task). Specifically, at the tool level, first, the data exchange tasks are configured through a visual Web operation interface and stored in the database. Then, after the task scheduling starts and before the data processing begins, the task sharding function code program of the main node will use SQL to count and sort out the data volume to be processed in this task, and reasonably split the data according to the processed data volume (such as by adopting an average distribution method, etc.), splitting the data into specific and non-repeating task data segments. Finally, using multi-threading technology, the split tasks are distributed to different sub-node services, enabling these sub-node services to concurrently execute the corresponding data upload tasks.

[0083] After the data upload module 3 uploads the relevant data to the business library 4, the business library 4 will normalize these data (i.e., the original data and compressed data whose data volume is less than or equal to the preset value). Since the medical and health data is huge in scale, and has many sources and types, the normalization process is to establish a standardized model specification according to the definition, format, attribute range, and relationship with other data of the data itself, and then organize it into a table form to form the business library 4. For example, for the basic information of patients (name, age, gender, etc.) and medical records (symptom description, medication, test results, etc.) in the medical and health data, their formats, value ranges, etc. will be determined according to unified standards, and then organized into corresponding table structures, so that the data is more standardized and orderly, which is convenient for subsequent use and management.

[0084] After the normalization processing of the business library 4, valid data is obtained. These valid data will be stored in the cache library 5. The role of the cache library 5 is to respond to subsequent data call requirements more quickly. For example, when it is necessary to query specific medical and health data for analysis or provide sharing services in the future, data can be obtained from the cache library 5 first, improving the efficiency of data use and ensuring the continuity and stability of medical and health data sharing services. At the same time, in order to better control the quality and amount of data, the system also supports setting a unified data increment threshold through the interface, or the system automatically calculates the increment threshold based on the frequency of the most recent data upload. If the threshold is exceeded, an active alarm will be issued. On the one hand, it can be actively fused, and on the other hand, it can remind the operation and maintenance personnel to manually fuse. You can also configure the validity period of the data, data volume threshold, data anomaly rules, etc. in the interface, actively limit data release and clean up the published historical data, and further ensure that the data stored in the cache library 5 is always valid and meets the requirements.

[0085] In addition, the client generates a corresponding data access request based on its own business needs, such as the need to query specific medical and health data (which may be the statistical information of patients with a certain disease, the detailed diagnosis and treatment records of a patient, etc.). This request contains key identification information of the data to be obtained, such as the data category, range, specific filtering conditions, etc., so that the server cluster can accurately locate the target data.

[0086] When the client initiates a request, the load layer will receive the request and perform preliminary verification on the format and legality of the request. For example, it checks whether the request conforms to the established communication protocol format and whether it contains necessary access parameters, etc., to ensure that the request is a valid request that can be correctly processed by the subsequent server cluster. After verification, the load layer sends the data access request to the server cluster accurately according to the preset communication link and rules.

[0087] After the server cluster receives a data access request from the load layer, it first parses the request to extract key information about the target data, such as the table, fields where the data to be queried is located, and the corresponding filtering conditions. This step is similar to accurately locating a specific location based on coordinates and feature descriptions on a map.

[0088] Based on the parsed key information, the server cluster performs a lookup operation in the cache library. Through technical means such as database query statements and indexes, it quickly locates the target data that meets the request requirements. Once the target data is located, the server cluster extracts this data from the cache library and prepares for the next sending operation.

[0089] During this process, the server cluster also performs some necessary processing on the obtained data, such as data format conversion (ensuring that the data format sent is consistent with the format expected by the client to receive), data encryption (if data security requirements are involved), etc., to ensure the accuracy and security of the data.

[0090] The server cluster sends the processed target data back to the load layer. After the load layer receives the target data sent back by the server cluster, it will again verify the integrity of the data, check whether it matches the checksum and other information added during sending, and ensure that there are no problems with the data during transmission.

[0091] After verification, the load layer accurately sends the target data to the client according to the client's address information, etc. After receiving the target data, the client can perform subsequent processing and display of this data according to its own business logic, such as displaying the queried patient diagnosis and treatment data on the interface of the medical system, generating corresponding statistical reports, etc., to complete the entire data access process.

[0092] The embodiment of the present invention also provides a computer device having the above Figure 1 shown high-concurrency-based medical and health data sharing service system.

[0093] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of a computer device provided by an optional embodiment of the present invention, as Figure 4As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting the components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 3 Taking one processor 10 as an example in

[0094] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field programmable gate array, a generic array logic, or any combination thereof.

[0095] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0096] The memory 20 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device presented by a kind of small program landing page, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 can optionally include a memory remotely set relative to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0097] The memory 20 can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 can also include a combination of the above types of memories.

[0098] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or communication networks.

[0099] Embodiments of the present invention also provide a computer-readable storage medium. The methods according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the methods described herein can be stored as such software processes on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0100] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A high-concurrency medical health data sharing service system, characterized in that: The system includes: a data processing device, a load layer and a server cluster, wherein the data processing device includes: a data upload interface, a front-end processor, a data upload module, a business library and a cache library; The data upload interface is used to receive raw data from different channels, and upload the raw data of each channel to the front-end processor respectively; for raw data with a data volume greater than a preset value, the data upload interface compresses the raw data, and uploads the compressed data to the front-end processor; The data uploading module is used to upload the data in the front-end processor to the service library in a distributed manner, and the service library is used to normalize the data to obtain valid data; The cache library is used to store the valid data; The load layer is used to receive a data access request from a client and send the data access request to the server cluster; The server cluster is used to obtain target data from the cache library according to the data access request, send the target data to the load layer, and send the target data to the client through the load layer.

2. The system according to claim 1, characterized in that The cache library includes: a first-level cache library and a second-level cache library, wherein: The first-level cache library is used to store the valid data; The secondary cache library is used to store hot data, which is valid data with an access frequency greater than a preset frequency; the access level of the secondary cache library is higher than that of the primary cache library.

3. The system according to claim 1, characterized in that The load layer includes: soft load and hard load, wherein: One hard load corresponds to multiple soft loads, wherein the hard load is oriented to the client; and the soft load is oriented to the server cluster.

4. The system according to claim 1, characterized in that The data upload module includes a main node and multiple sub-nodes, among which: The master node is used to split the data in the front-end according to the data volume, and form multiple tasks according to the split results, and send each task to each child node respectively; The child node is used to execute the task and write the data corresponding to the task into the business library.

5. The system according to claim 1, characterized in that include: If the data type of the data is structured data, the data upload interface uploads the data to the relational database deployed on the front-end processor.

6. The system according to claim 1, characterized in that include: If the data type of the data is unstructured data, the data is uploaded to a file server.

7. The system according to claim 1, characterized in that include: The business library is also used to determine the collection time of each data, and delete the data that exceeds the preset time limit according to the collection time of each data.

8. The system according to claim 1, characterized in that include: Each channel is set with a preset daily settlement capacity. If the amount of data provided by a channel is greater than the preset daily capacity, the data upload interface will fuse the channel. After the channel is blown, the channel will no longer transmit data on that day.

9. A medical health data sharing method based on high concurrency, characterized in that: The method comprises: Receive raw data from different channels; Compare the data volume corresponding to the data of each channel with a preset value, and compress the original data whose data volume is greater than the preset value to obtain compressed data; The original data whose data volume is less than or equal to the preset value and the compressed data are uploaded to the business library in a distributed manner, so that the business library normalizes the original data whose data volume is less than or equal to the preset value and the compressed data to obtain valid data, and stores the valid data in the cache library.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the high-concurrency medical and health data sharing method described in claim 9 by executing the computer instructions.

Images