Health big data platform architecture optimization method and system based on cloud computing

Abstract

The invention discloses a health big data platform architecture optimization method and system based on cloud computing, and relates to the technical field of electric digital data processing. Firstly, the data transmission process of the health big data platform is analyzed to obtain data transmission optimization information of the health big data platform, and then data transmission architecture optimization configuration is performed, so that the data transmission efficiency and stability can be improved, and the risks of transmission delay and data loss are reduced; then, dividing the health data after data transmission optimization configuration to obtain each health data storage fragment, and further performing storage architecture modulation on the data of each health data storage fragment of the health big data platform, thereby being beneficial to optimizing the data storage performance, improving the data query speed and the response capability of the platform, and improving the user experience. And finally, the task processing process of each task processing node is monitored and analyzed, and then elastic computing resource optimization management is performed on the health big data platform, so that the resource utilization rate can be improved, and the overall performance of the platform is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a method and system for optimizing a health big data platform architecture based on cloud computing. Background Art

[0002] In the field of health big data platform, with the rapid development of information technology and the growth of medical and health data, efficient and secure storage, processing and analysis of massive health data has become one of the key technologies to improve the quality of medical services and health management level. Health big data covers all kinds of data from individual physiology, diseases to environmental factors, and is diverse, high-dimensional and complex. The health big data platform based on cloud computing realizes distributed storage and real-time processing of data through the powerful computing and storage capabilities of cloud computing, and can conduct in-depth mining and intelligent prediction of health data, thereby promoting the intelligentization and modernization of medical and health services.

[0003] Prior art, such as an automatic optimization method for the configuration of a big data processing platform, is disclosed in the invention patent publication with publication number: CN113032033B. The method collects a configuration parameter training set according to the following steps: randomly generating a set of configuration parameters for the big data processing platform, running the set of configuration parameters, and monitoring the execution time t, and terminating the operation when the execution time t exceeds the dynamically set maximum permissible time Tmax; during the operation of the configuration parameters, determining whether to exit the process of collecting the configuration parameter training set according to the fluctuation of the execution time t; after deciding to exit the process of collecting the configuration parameter training set, selecting a set of configuration parameters with the shortest execution time from all successfully run configuration parameters as the optimal configuration parameters.

[0004] Prior art, such as the invention patent announcement with announcement number: CN110046143B, discloses an overall architecture optimization system and optimization method for an integrated data platform, including a database, a database operation status monitoring module electrically connected to the database operation status monitoring module, a database operation status monitoring module electrically connected to the management end and an optimization scheme selection module, a hard disk and a server are both bidirectionally electrically connected to the allocation adjustment module, the allocation adjustment module is bidirectionally electrically connected to the optimization model generation module, an adjustment command confirmation module is electrically input connected to the management end, and the allocation adjustment module is electrically output connected to the process optimization module, and the process optimization module is electrically output connected to the server.

[0005] Combined with the above solutions, it is found that currently in the field of health big data architecture optimization, only the data collection process and module optimization process are usually analyzed. However, in the data storage process, there will be problems such as the inability to perform refined management due to excessive data volume and unreasonable storage structure. This not only affects the efficiency and stability of data transmission and storage, but may also create the risk of resource waste, affecting the operating efficiency and data service quality of the health big data platform, and thus affecting resource utilization. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a method and system for optimizing the architecture of a health big data platform based on cloud computing, which can effectively solve the problems involved in the above-mentioned background technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cloud computing-based health big data platform architecture optimization method, including analyzing the data transmission process of the health big data platform, obtaining data transmission optimization information of the health big data platform, and optimizing the data transmission architecture of the health big data platform based on the data transmission optimization information of the health big data platform.

[0008] The health data after the statistical data transmission optimization configuration is divided into several segments according to time, which are recorded as each health data storage segment.

[0009] The query data of each health data storage segment is analyzed, and then the storage architecture of each health data storage segment of the health big data platform is modulated.

[0010] After the statistical storage architecture is modulated, each task node of the health big data platform is recorded as a task processing node. The task processing process of each task processing node is monitored and analyzed, and then the elastic computing resources of the health big data platform are optimized and managed.

[0011] The second aspect of the present invention provides a health big data platform architecture optimization system based on cloud computing, including: a data transmission architecture optimization module, which is used to analyze the data transmission process of the health big data platform, obtain data transmission optimization information of the health big data platform, and optimize the data transmission architecture configuration of the health big data platform based on the data transmission optimization information of the health big data platform.

[0012] The data storage partitioning module is used to store the health data after the data transmission optimization configuration is optimized, and divide it into several segments according to time, which are recorded as each health data storage segment.

[0013] The storage architecture modulation module is used to analyze the query data of each health data storage segment, and then perform storage architecture modulation on the data of each health data storage segment of the health big data platform.

[0014] The computing resource optimization management module is used to count the task nodes of the health big data platform after the storage architecture is modulated, recorded as task processing nodes, monitor and analyze the task processing process of each task processing node, and then optimize the elastic computing resources of the health big data platform.

[0015] The present invention has the following beneficial effects: (1) The present invention provides a method for optimizing the architecture of a health big data platform based on cloud computing. First, the data transmission process is analyzed, and then the data transmission architecture is optimized and configured, which can reduce service interruptions caused by transmission problems. Then, the storage architecture of each health data storage fragment is modulated, which helps to improve the flexibility of data storage and reduce storage costs. Finally, elastic computing resources are optimized and managed, and resources can be dynamically adjusted according to actual needs to improve resource utilization.

[0016] (2) The present invention optimizes the data transmission architecture of the health big data platform according to the data transmission optimization information of the health big data platform, which helps to improve the efficiency of data transmission, reduce delays and packet loss during data transmission, enhance the stability and reliability of data transmission, and thus improve the accuracy of health data. At the same time, the optimized data transmission architecture can improve the platform's ability to handle high-concurrency data and reduce the risk of system failures caused by data transmission problems.

[0017] (3) The present invention analyzes the query data of each health data storage segment and then modulates the storage architecture of each health data storage segment of the health big data platform, thereby improving the efficiency of data storage and retrieval. Through the refined management and optimization of the storage segments, the response time of data queries is reduced, the overall performance of the platform is improved, and at the same time, the utilization rate of storage resources is optimized and the cost of data storage is reduced.

[0018] (4) The present invention monitors and analyzes the task processing process of each task processing node, and then optimizes the management of elastic computing resources of the health big data platform, thereby effectively improving the resource scheduling capability of the platform and ensuring that the platform can dynamically adjust computing resources to adapt to changing needs during data peak periods and when task loads change. This not only improves resource utilization, but also enhances the stability and scalability of the platform, ensuring the continuity and efficiency of task processing.

[0019] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0021] Figure 2 It is a schematic diagram of system module connection of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] See also Figure 1 As shown, the first aspect of an embodiment of the present invention provides a technical solution: a method for optimizing the architecture of a health big data platform based on cloud computing, comprising analyzing the data transmission process of the health big data platform, obtaining data transmission optimization information of the health big data platform, and optimizing the data transmission architecture configuration of the health big data platform based on the data transmission optimization information of the health big data platform.

[0024] It should be noted that the health big data platform is a comprehensive system that integrates the collection, storage, processing, analysis and sharing of health-related information.

[0025] The health data after the statistical data transmission optimization configuration is divided into several segments according to time, which are recorded as each health data storage segment.

[0026] The query data of each health data storage segment is analyzed, and then the storage architecture of each health data storage segment of the health big data platform is modulated.

[0027] After the statistical storage architecture is modulated, each task node of the health big data platform is recorded as a task processing node. The task processing process of each task processing node is monitored and analyzed, and then the elastic computing resources of the health big data platform are optimized and managed.

[0028] Specifically, the data transmission process of the health big data platform is analyzed, and the specific analysis process is: the data transmission process of the health big data platform is analyzed to obtain the data transmission parameters of the health big data platform, and the data transmission parameters of the health big data platform include the average data transmission delay duration of the health big data platform within a preset period, the maximum data transmission coefficient of the network link, the average utilization rate of the data packet payload, the data transmission anomaly coefficient and the average data transmission rate.

[0029] The data transmission efficiency evaluation value of the health big data platform is obtained based on the data transmission parameter processing of the health big data platform, and the data transmission efficiency evaluation value of the health big data platform is used to comprehensively quantify the overall performance of data transmission.

[0030] It should be noted that the average data transmission delay refers to the average time it takes for data to travel from the sender to the receiver. The delay of each data packet is recorded through network analysis tools (such as Wireshark and TCPdump), the delay of all data packets is summed up, and then divided by the total number of data packets to obtain the average data transmission delay. The maximum data transmission coefficient of a network link refers to the ratio of the maximum data transmission rate that a network link can achieve within a preset period to the theoretical maximum transmission rate. Network bandwidth test tools (such as iperf and Speedtest.net) can be used to measure the actual maximum transmission rate of a network link within a preset period, and then the maximum data transmission coefficient of a network link can be obtained. The average data packet payload utilization refers to the ratio of the maximum data transmission rate of each data packet within a preset period. According to the average value of the ratio of the payload (actual data) in the data packet to the total data packet size, the network analysis tool can be used to record the payload of each data packet and the total size of the data packet, and then the payload utilization of each data packet can be obtained, and the average utilization of the data packet payload can be obtained by averaging. The data transmission anomaly coefficient reflects the abnormality (such as data packet loss, error) in the data transmission process within the preset period. Use network monitoring tools (such as Nagios, Zabbix) to count the number of data transmission anomaly events that occur within the preset period, and remove the unit of the ratio of the number of data transmission anomaly events to the total number of transmissions to obtain the data transmission anomaly coefficient. The network bandwidth test tool can be used to measure the average data transmission rate of the health big data platform within the preset period.

[0031] Specifically, the data transmission efficiency evaluation value of the health big data platform is analyzed under the following conditions: ; In the formula, represents the data transmission efficiency evaluation value of the health big data platform, Indicates the average data transmission delay of the health big data platform. Indicates the data transmission performance correction factor corresponding to the set average delay time of unit data transmission. represents the maximum data transmission coefficient of the network link of the health big data platform, Indicates the data transmission efficiency correction factor corresponding to the set maximum data transmission coefficient of the network link. represents the average utilization of data packet payload of the health big data platform, Indicates the data transmission efficiency correction factor corresponding to the set average utilization rate of the unit data packet payload. represents the data transmission anomaly coefficient of the health big data platform, Indicates the data transmission efficiency correction factor corresponding to the set data transmission abnormality coefficient. represents the average data transmission rate of the health big data platform, It represents the data transmission efficiency correction factor corresponding to the set unit data transmission rate, and e represents a natural constant.

[0032] It should be added that, in the present embodiment, the data transmission efficiency correction factor corresponding to the preset unit data transmission average delay time, the data transmission efficiency correction factor corresponding to the maximum data transmission coefficient of the network link, the data transmission efficiency correction factor corresponding to the unit data packet payload average utilization rate, the data transmission efficiency correction factor corresponding to the data transmission anomaly coefficient and the data transmission efficiency correction factor corresponding to the unit data transmission rate are obtained from the platform architecture optimization database.

[0033] It needs to be explained that the data transmission efficiency correction factors corresponding to the average data transmission delay time per unit, the maximum data transmission coefficient of the network link, the average utilization rate of the data packet payload, the data transmission anomaly coefficient and the unit data transmission rate are respectively used to adjust the importance of the average data transmission delay time, the maximum data transmission coefficient of the network link, the average utilization rate of the data packet payload, the data transmission anomaly coefficient and the average data transmission rate of the health big data platform in the process of analyzing and obtaining the data transmission efficiency evaluation value of the health big data platform. For example, there is a pre-set mapping relationship between the data transmission parameters of the health big data platform and the corresponding data transmission efficiency correction factors in the platform architecture optimization database. The pre-set mapping relationship can be used to match the data transmission efficiency correction factors corresponding to the data transmission parameters of the real-time health big data platform. The average data transmission delay time, the maximum data transmission coefficient of the network link, the average utilization rate of the data packet payload, the data transmission anomaly coefficient and the average data transmission rate of the health big data platform are matched with the pre-set mapping relationship respectively, and the data transmission efficiency correction factors corresponding to the average data transmission delay time, the maximum data transmission coefficient of the network link, the average utilization rate of the data packet payload, the data transmission anomaly coefficient and the unit data transmission rate are obtained.

[0034] In this implementation scheme, there is a correlation between the average data transmission delay time, the maximum data transmission coefficient of the network link, the average utilization rate of the data packet payload, the data transmission anomaly coefficient and the average data transmission rate of the health big data platform, and they do not exist independently. For example, if the maximum data transmission coefficient of the network link is low, even if the average data transmission delay time is very low, the average data transmission rate will also decrease. If the data transmission anomaly coefficient is high, it means that the network link is unstable and there are many errors, packet losses or retries. Each retransmission will increase the average data transmission delay time and may reduce the actual maximum transmission rate, resulting in the maximum data transmission coefficient of the network link being affected. The low average utilization rate of the data packet payload may increase the burden on the network link, increase the incidence of anomalies, and increase the data transmission anomaly coefficient. Comprehensive analysis to obtain the data transmission efficiency evaluation value of the health big data platform can help identify the data transmission performance bottleneck of the health big data platform, understand the maximum transmission capacity and actual utilization rate of the network link, and thus more effectively allocate network resources.

[0035] Specifically, the data transmission optimization information of the health big data platform is obtained, and the specific process is: the data transmission optimization information of the health big data platform includes executing data transmission optimization and not executing data transmission optimization.

[0036] The data transmission efficiency evaluation value of the health big data platform is compared with the set data transmission efficiency evaluation threshold. If the data transmission efficiency evaluation value of the health big data platform is higher than or equal to the set data transmission efficiency evaluation threshold, the data transmission optimization information of the health big data platform is marked as not performing data transmission optimization; otherwise, the data transmission optimization information of the health big data platform is marked as performing data transmission optimization.

[0037] Specifically, the data transmission architecture of the health big data platform is optimized and configured based on the data transmission optimization information of the health big data platform. The specific process is: extract the data transmission optimization information of the health big data platform. If the data transmission optimization information of the health big data platform is not to perform data transmission optimization, then continue to optimize the data transmission architecture configuration with the current data transmission compression ratio.

[0038] If the data transmission optimization information of the health big data platform is to perform data transmission optimization, the current data transmission compression ratio is added to the set data transmission compression ratio correction value to obtain the target data transmission compression ratio, and the data transmission compression ratio of the health big data platform is adjusted to the target data transmission compression ratio to perform data transmission architecture optimization configuration.

[0039] It should be noted that analyzing the data transmission process of the health big data platform can reflect the efficiency, stability and reliability of data transmission. If the data transmission performance is unqualified and subsequent data transmission is carried out with the original default data transmission compression ratio, it may lead to increased data transmission delays, reduced data transmission quality, and even the risk of data transmission interruption or data loss. Therefore, it is necessary to optimize the data transmission architecture configuration in combination with the data transmission compression ratio correction value to ensure the smoothness of data transmission and the accuracy of data processing, so as to provide data support for optimizing the overall data transmission performance of the health big data platform.

[0040] Specifically, the query data of each health data storage segment is analyzed. The specific analysis process is: based on the data transmission efficiency evaluation value of the health big data platform, the average data transmission efficiency evaluation value of the health big data platform is statistically obtained, and the data storage evaluation influencing factor is obtained according to the matching of the average data transmission efficiency evaluation value of the health big data platform.

[0041] It needs to be explained that the average evaluation value of the data transmission efficiency of the health big data platform refers to the average value of the data transmission efficiency evaluation value of the health big data platform within each preset period. The data storage evaluation influencing factor is obtained by matching the average evaluation value of the data transmission efficiency of the health big data platform. The specific process is to match the average evaluation value of the data transmission efficiency of the health big data platform with the data storage evaluation influencing factors corresponding to each data transmission efficiency average evaluation value interval stored in the platform architecture optimization database, and count the data storage evaluation influencing factors corresponding to the interval in which the average evaluation value of the data transmission efficiency of the health big data platform is located, which are recorded as data storage evaluation influencing factors. The larger the average evaluation value of the data transmission efficiency of the health big data platform, the better the quality of the data transmission process, and the smaller the impact on subsequent storage evaluation. Therefore, the smaller the data storage evaluation influencing factor obtained by matching.

[0042] The query data of each health data storage segment includes the query frequency, query response time fluctuation coefficient, query failure rate and query concurrency coefficient of each health data storage segment within a preset period.

[0043] It should be noted that the query frequency of each health data storage fragment refers to the number of times each health data storage fragment is queried within a preset period. The number of queries for each health data storage fragment within the preset period can be obtained by monitoring log statistics, and the ratio of the number of queries to the preset period length is counted as the query frequency of each health data storage fragment. The query response time fluctuation coefficient is used to measure the stability of the query response time. The database performance monitoring tool is used to measure the response time of each query within the preset period, and the response time is averaged to obtain the average response time. The squares of the differences between the response time of each query and the average response time are summed and then divided by the total number of queries. The square root is then taken to obtain the standard deviation. The obtained standard deviation The query response time fluctuation coefficient is obtained by removing the unit from the ratio with the average response time. The query failure rate refers to the ratio of the number of failed query attempts to the total number of queries within a preset period. The total number of queries and the number of failed queries within the preset period can be obtained by monitoring log statistics. The ratio of the number of failed queries to the total number of queries is recorded as the query failure rate. The query concurrency coefficient is an indicator that measures the ability of a data storage fragment to process multiple query requests within a preset period. The time points of all query requests are recorded by monitoring logs within the preset period, and the time point when the number of query requests occurring simultaneously within the preset period reaches the maximum is found. The query concurrency coefficient is the value of the number of concurrent requests at that time point after removing the unit from the ratio.

[0044] The query data of each healthy data storage segment is comprehensively analyzed to obtain a data storage characterization value of each healthy data storage segment, and the data storage characterization value of each healthy data storage segment is used to comprehensively quantify the query status of the data storage segment.

[0045] In this embodiment, the data storage characterization value of each healthy data storage segment can be obtained by the following analysis method, and the specific analysis conditions are as follows: ; In the formula, The data storage representation value representing the i-th healthy data storage segment, represents the query frequency of the i-th health data storage segment, represents the defined query frequency of the i-th health data storage segment, represents the query response time fluctuation coefficient of the i-th health data storage segment, Indicates the data storage evaluation factor corresponding to the set query response time fluctuation coefficient. represents the query failure rate of the i-th health data storage segment, Indicates the data storage evaluation factor corresponding to the set unit query failure rate. represents the query concurrency coefficient of the i-th health data storage segment, represents the bounded query concurrency coefficient of the i-th health data storage segment, i represents the number of each health data storage segment, , n represents the total number of healthy data storage fragments.

[0046] It should be added that, in this embodiment, a data storage assessment factor corresponding to a preset query response time fluctuation coefficient and a data storage assessment factor corresponding to a unit query failure rate are obtained from the platform architecture optimization database.

[0047] It needs to be explained that the data storage evaluation factors corresponding to the query response time fluctuation coefficient and the unit query failure rate are respectively used to adjust the importance of the query response time fluctuation coefficient and the query failure rate of each health data storage fragment in the process of analyzing and obtaining the data storage characterization value of each health data storage fragment. For example, there is a pre-set mapping relationship between the query data of each health data storage fragment and the corresponding data storage evaluation factor in the platform architecture optimization database. The pre-set mapping relationship can be used to match the real-time data storage evaluation factor corresponding to the query data of each health data storage fragment, and the query response time fluctuation coefficient and the query failure rate of each health data storage fragment are respectively matched with the pre-set mapping relationship to obtain the query response time fluctuation coefficient and the data storage evaluation factor corresponding to the unit query failure rate.

[0048] In this implementation scheme, the query frequency, query response time fluctuation coefficient, query failure rate and query concurrency coefficient of each healthy data storage fragment are correlated and do not exist independently. For example, a high query frequency usually increases the load of the platform, especially when the query requests are concentrated. The platform may not be able to process all query requests in time, resulting in unstable response time, which leads to an increase in the query response time fluctuation coefficient. It may also cause query timeout, database connection failure or query request discarded, thereby increasing the query failure rate. A large query response time fluctuation coefficient is often a manifestation of uneven platform load, insufficient resources or bottlenecks, which may cause query failures. When the response time is too long, the query may timeout or be interrupted, and the query failure rate will increase. When the query concurrency is too high, resource competition may occur, resulting in unstable response time, thereby increasing the query response time fluctuation coefficient. Comprehensive analysis to obtain the data storage characterization value of each healthy data storage fragment can more accurately evaluate the utilization status of the healthy data storage fragment and make a more reasonable allocation of storage resources, thereby improving the response efficiency of the overall storage system.

[0049] Based on the data storage characterization value of each healthy data storage fragment and combined with the data storage evaluation influencing factor, comprehensive processing is performed to obtain the data storage comprehensive index value of each healthy data storage fragment, and the data storage comprehensive index value of each healthy data storage fragment is used to comprehensively quantify the query resource utilization status of the data storage fragment.

[0050] It should be supplemented that the product of the data storage characterization value of each healthy data storage segment and the data storage evaluation impact factor is recorded as the data storage comprehensive index value of each healthy data storage segment.

[0051] Specifically, the storage architecture of the data of each health data storage fragment of the health big data platform is modulated. The specific process is: the data storage comprehensive index value of each health data storage fragment is compared with the set data storage comprehensive index threshold. If the data storage comprehensive index value of a health data storage fragment is higher than or equal to the set data storage comprehensive index threshold, the health data storage fragment is marked as a health data high-frequency query storage fragment, thereby traversing all health data high-frequency query storage fragments, and transmitting the data in each health data high-frequency query storage fragment to the high-performance cloud computing service platform.

[0052] If the data storage comprehensive index value of a healthy data storage segment is lower than the set data storage comprehensive index threshold, the healthy data storage segment will be marked as a healthy data low-frequency query storage segment, thereby traversing all healthy data low-frequency query storage segments, and transmitting the data in each healthy data low-frequency query storage segment to the normal performance storage service platform.

[0053] It should be added that the high-performance cloud computing service platform refers to a cloud storage service that provides fast data access, high throughput, and low latency, and the normal performance storage service platform refers to a cloud storage service that meets general business needs and has relatively low costs.

[0054] It needs to be explained that placing frequently queried healthy data storage fragments on a high-performance cloud computing service platform can ensure that the data can be retrieved and accessed more quickly, thereby improving the overall data query efficiency. By storing data separately according to query frequency, storage resources can be allocated more reasonably, avoiding wasting resources on high-performance storage platforms to store infrequently accessed data, while effectively utilizing resources on normal performance storage service platforms.

[0055] Specifically, the task processing process of each task processing node is monitored and analyzed. The specific analysis process is: the task processing process of each task processing node is monitored and analyzed to obtain the task processing data of each task processing node. The task processing data of each task processing node includes the task execution time fluctuation coefficient, task failure rate, resource saturation and average task execution time of each task processing node within a preset period.

[0056] It should be noted that the task execution time fluctuation coefficient reflects the stability of the task processing time. System monitoring tools (such as Prometheus and Grafana) can be used to record the execution time of each task within the preset period, and the average execution time can be obtained by averaging. The square of the difference between each task execution time and the average execution time is summed to obtain a value. This value is divided by the number of tasks and then squared to obtain the standard deviation of the task execution time. The value of the standard deviation of the task execution time divided by the average execution time is recorded as the task execution time fluctuation coefficient. The task failure rate refers to the ratio of the number of task execution failures to the total number of executions within the preset period. The total number of executions and failures of tasks within the preset period can be monitored and recorded through log analysis tools (such as ELK Stack and Splunk). The ratio of the number of failures to the total number of executions is recorded as the task failure rate. Resource saturation refers to the degree of use of task processing node resources (such as CPU, memory, and disk I / O). The platform's built-in monitoring tools are used to monitor the resource usage and total resource volume of the task processing node within the preset period, and the ratio of resource usage to total resource volume is recorded as resource saturation.

[0057] A basic evaluation value of task processing of each task processing node is obtained based on comprehensive analysis of task processing data of each task processing node, and the basic evaluation value of task processing of each task processing node is used to comprehensively quantify the task processing load of each task processing node.

[0058] In this embodiment, the task processing basic evaluation value of each task processing node can be obtained by the following analysis method, and the specific analysis conditions are as follows: ; In the formula, represents the basic evaluation value of task processing of the jth task processing node, represents the task execution time fluctuation coefficient of the jth task processing node, It represents the task processing efficiency evaluation factor corresponding to the set task execution time fluctuation coefficient. represents the task failure rate of the jth task processing node, It represents the task processing efficiency evaluation factor corresponding to the set unit task failure rate. represents the resource saturation of the jth task processing node, represents the defined resource saturation of the jth task processing node, represents the average execution time of the task of the jth task processing node, represents the defined task execution time of the jth task processing node, j represents the number of each task processing node, , m represents the total number of task processing nodes.

[0059] It should be added that, in this embodiment, the task processing efficiency evaluation factor corresponding to the preset task execution time fluctuation coefficient and the task processing efficiency evaluation factor corresponding to the unit task failure rate are obtained from the platform architecture optimization database.

[0060] It needs to be explained that the task execution time fluctuation coefficient and the task failure rate corresponding to the task processing efficiency evaluation factors are respectively used to adjust the importance of the task execution time fluctuation coefficient and the task failure rate of each task processing node in the process of analyzing and obtaining the task processing basic evaluation value of each task processing node. For example, there is a preset mapping relationship between the task processing data of each task processing node and the corresponding task processing efficiency evaluation factor in the platform architecture optimization database. The task processing efficiency evaluation factors corresponding to the real-time task processing data of each task processing node can be matched through the preset mapping relationship. The task execution time fluctuation coefficient and the task failure rate of each task processing node are respectively matched with the preset mapping relationship to obtain the task processing efficiency evaluation factors corresponding to the task execution time fluctuation coefficient and the unit task failure rate.

[0061] In this implementation scheme, the task execution time fluctuation coefficient, task failure rate, resource saturation and average task execution time of each task processing node are correlated and do not exist independently. For example, when the task execution time fluctuation coefficient is large, the task execution time is unstable, which may cause the task execution time to exceed the average value, thereby increasing the average task execution time. When the platform resources are close to saturation, the task may not be able to obtain sufficient resources in time to complete the execution, resulting in task failure. Therefore, a higher resource saturation is usually accompanied by a higher task failure rate. A large task execution time fluctuation coefficient usually increases the risk of task failure. Long-term fluctuations in task execution time may cause task timeouts, thereby causing task failure. A high task failure rate may mean that the platform's resource allocation is unreasonable, resulting in increased time consumption during task execution, thereby lengthening the average task execution time. Comprehensive analysis to obtain the basic evaluation value of task processing for each task processing node can help identify the key factors affecting task processing performance, thereby improving the overall efficiency of task processing.

[0062] Specifically, the elastic computing resources of the health big data platform are optimized and managed, and the specific process is: extract the task processing basic evaluation value of each task processing node, compare the task processing basic evaluation value of each task processing node with the set task processing basic first evaluation threshold, if the task processing basic evaluation value of a task processing node is higher than or equal to the set task processing basic first evaluation threshold, then mark the task processing node as a high task volume processing node.

[0063] If the task processing basic evaluation value of a task processing node is lower than the set task processing basic first evaluation threshold, the task processing basic evaluation value of the task processing node is compared with the set task processing basic second evaluation threshold; if the task processing basic evaluation value of the task processing node is higher than or equal to the set task processing basic second evaluation threshold, the task processing node is marked as a normal task volume processing node.

[0064] If the task processing basic evaluation value of the task processing node is lower than the set task processing basic second evaluation threshold, the task processing node is marked as a low task volume processing node.

[0065] Elastic computing resource optimization management is performed on task processing nodes marked as high-task-load processing nodes and low-task-load processing nodes.

[0066] It should be noted that elastic computing resource optimization management is performed on task processing nodes marked as high-task volume processing nodes and low-task volume processing nodes. The specific process is to send prompt information to the management terminal, prompting the high-task volume processing nodes to add data processing nodes, increase computing resources (such as increasing CPU, memory and other resources), and improve network bandwidth; prompting the low-task volume processing nodes to reduce data processing nodes, reduce computing resources (such as reducing CPU, memory and other resources), and reduce network bandwidth. At the same time, the load balancer is used to automatically assign new tasks to the low-task volume processing nodes, which helps to improve resource utilization efficiency, reduce platform operating costs, and ensure that the platform maintains high performance and stability when facing fluctuating loads.

[0067] like Figure 2 The second aspect of the present invention provides a health big data platform architecture optimization system based on cloud computing, including: a data transmission architecture optimization module, which is used to analyze the data transmission process of the health big data platform, obtain the data transmission optimization information of the health big data platform, and optimize the data transmission architecture configuration of the health big data platform based on the data transmission optimization information of the health big data platform.

[0068] The data storage partitioning module is used to store the health data after the data transmission optimization configuration is optimized, and divide it into several segments according to time, which are recorded as each health data storage segment.

[0069] The storage architecture modulation module is used to analyze the query data of each health data storage segment, and then perform storage architecture modulation on the data of each health data storage segment of the health big data platform.

[0070] The computing resource optimization management module is used to count the task nodes of the health big data platform after the storage architecture is modulated, recorded as task processing nodes, monitor and analyze the task processing process of each task processing node, and then optimize the elastic computing resource management of the health big data platform.

[0071] It should be noted that the cloud computing-based health big data platform architecture optimization system also includes a platform architecture optimization database for storing the first parameter set, the second parameter set and the third parameter set obtained by analyzing historical data.

[0072] The first parameter set includes the data transmission efficiency correction factor corresponding to the average delay time of unit data transmission, the data transmission efficiency correction factor corresponding to the maximum data transmission coefficient of the network link, the data transmission efficiency correction factor corresponding to the average utilization rate of the unit data packet payload, the data transmission efficiency correction factor corresponding to the data transmission anomaly coefficient, the data transmission efficiency correction factor corresponding to the unit data transmission rate, the data transmission efficiency evaluation threshold and the data transmission compression ratio correction value.

[0073] The second parameter set includes the data storage assessment impact factor corresponding to each data transmission efficiency average assessment value interval, the defined query frequency of each healthy data storage segment, the data storage assessment factor corresponding to the query response time fluctuation coefficient, the data storage assessment factor corresponding to the unit query failure rate, the defined query concurrency coefficient of each healthy data storage segment and the data storage comprehensive indicator threshold.

[0074] The third parameter set includes the task processing efficiency evaluation factor corresponding to the task execution time fluctuation coefficient, the task processing efficiency evaluation factor corresponding to the unit task failure rate, the defined resource saturation of each task processing node, the defined task execution time of each task processing node, the first evaluation threshold of the task processing basis and the second evaluation threshold of the task processing basis.

[0075] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0076] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well.

Claims

1. A method for optimizing the architecture of a health big data platform based on cloud computing, characterized in that: include: Analyze the data transmission process of the health big data platform, obtain the data transmission optimization information of the health big data platform, and optimize the data transmission architecture configuration of the health big data platform based on the data transmission optimization information of the health big data platform; The health data after the optimized configuration of statistical data transmission is divided into several segments according to time, which are recorded as each health data storage segment; Analyze the query data of each health data storage segment, and then modulate the storage architecture of each health data storage segment of the health big data platform; After the statistical storage architecture is modulated, each task node of the health big data platform is recorded as a task processing node. The task processing process of each task processing node is monitored and analyzed, and then the elastic computing resources of the health big data platform are optimized and managed.

2. The cloud computing-based health big data platform architecture optimization method according to claim 1 is characterized by: The data transmission process of the health big data platform is analyzed, and the specific analysis process is as follows: Analyze the data transmission process of the health big data platform to obtain the data transmission parameters of the health big data platform, wherein the data transmission parameters of the health big data platform include the average data transmission delay time of the health big data platform within a preset period, the maximum data transmission coefficient of the network link, the average utilization rate of the data packet payload, the data transmission abnormality coefficient and the average data transmission rate; The data transmission efficiency evaluation value of the health big data platform is obtained based on the data transmission parameter processing of the health big data platform, and the data transmission efficiency evaluation value of the health big data platform is used to comprehensively quantify the overall performance of data transmission.

3. The cloud computing-based health big data platform architecture optimization method according to claim 2 is characterized by: The specific process of obtaining the data transmission optimization information of the health big data platform is as follows: The data transmission optimization information of the health big data platform includes executing data transmission optimization and not executing data transmission optimization; The data transmission efficiency evaluation value of the health big data platform is compared with the set data transmission efficiency evaluation threshold. If the data transmission efficiency evaluation value of the health big data platform is higher than or equal to the set data transmission efficiency evaluation threshold, the data transmission optimization information of the health big data platform is marked as not performing data transmission optimization; otherwise, the data transmission optimization information of the health big data platform is marked as performing data transmission optimization.

4. The cloud computing-based health big data platform architecture optimization method according to claim 3 is characterized by: The data transmission optimization information based on the health big data platform is used to optimize the data transmission architecture of the health big data platform. The specific process is as follows: Extracting data transmission optimization information of the health big data platform, if the data transmission optimization information of the health big data platform indicates that data transmission optimization is not to be performed, continuing to perform data transmission architecture optimization configuration with the current data transmission compression ratio; If the data transmission optimization information of the health big data platform is to perform data transmission optimization, the current data transmission compression ratio is added to the set data transmission compression ratio correction value to obtain the target data transmission compression ratio, and the data transmission compression ratio of the health big data platform is adjusted to the target data transmission compression ratio to perform data transmission architecture optimization configuration.

5. The cloud computing-based health big data platform architecture optimization method according to claim 1, characterized in that: The query data of each health data storage segment is analyzed, and the specific analysis process is as follows: Based on the data transmission efficiency evaluation value of the health big data platform, the average data transmission efficiency evaluation value of the health big data platform is obtained by statistics, and the data storage evaluation influencing factor is obtained according to the matching of the average data transmission efficiency evaluation value of the health big data platform; The query data of each health data storage segment includes the query frequency, query response time fluctuation coefficient, query failure rate and query concurrency coefficient of each health data storage segment within a preset period; Comprehensively analyzing the query data of each health data storage segment to obtain a data storage characterization value of each health data storage segment, wherein the data storage characterization value of each health data storage segment is used to comprehensively quantify the query status of the data storage segment; Based on the data storage characterization value of each healthy data storage fragment and combined with the data storage evaluation influencing factor, comprehensive processing is performed to obtain the data storage comprehensive index value of each healthy data storage fragment, and the data storage comprehensive index value of each healthy data storage fragment is used to comprehensively quantify the query resource utilization status of the data storage fragment.

6. The cloud computing-based health big data platform architecture optimization method according to claim 5 is characterized by: The storage architecture modulation of the data of each health data storage fragment of the health big data platform is specifically performed as follows: The data storage comprehensive index value of each health data storage segment is compared with the set data storage comprehensive index threshold. If the data storage comprehensive index value of a health data storage segment is higher than or equal to the set data storage comprehensive index threshold, the health data storage segment is marked as a health data high-frequency query storage segment, thereby traversing all health data high-frequency query storage segments, and transmitting the data in each health data high-frequency query storage segment to the high-performance cloud computing service platform; If the data storage comprehensive index value of a healthy data storage segment is lower than the set data storage comprehensive index threshold, the healthy data storage segment will be marked as a healthy data low-frequency query storage segment, thereby traversing all healthy data low-frequency query storage segments, and transmitting the data in each healthy data low-frequency query storage segment to the normal performance storage service platform.

7. The cloud computing-based health big data platform architecture optimization method according to claim 1, characterized in that: The task processing process of each task processing node is monitored and analyzed, and the specific analysis process is as follows: Monitor and analyze the task processing process of each task processing node to obtain the task processing data of each task processing node; A basic evaluation value of task processing of each task processing node is obtained based on comprehensive analysis of task processing data of each task processing node, and the basic evaluation value of task processing of each task processing node is used to comprehensively quantify the task processing load of each task processing node.

8. The cloud computing-based health big data platform architecture optimization method according to claim 7, characterized in that: The specific process of optimizing the elastic computing resources of the health big data platform is as follows: Extracting the task processing basic evaluation value of each task processing node, comparing the task processing basic evaluation value of each task processing node with the set task processing basic first evaluation threshold, if the task processing basic evaluation value of a task processing node is higher than or equal to the set task processing basic first evaluation threshold, marking the task processing node as a high task volume processing node; If the task processing basic evaluation value of a task processing node is lower than the set task processing basic first evaluation threshold, the task processing basic evaluation value of the task processing node is compared with the set task processing basic second evaluation threshold; if the task processing basic evaluation value of the task processing node is higher than or equal to the set task processing basic second evaluation threshold, the task processing node is marked as a normal task volume processing node; If the task processing basic evaluation value of the task processing node is lower than the set task processing basic second evaluation threshold, the task processing node is marked as a low task processing node; Elastic computing resource optimization management is performed on task processing nodes marked as high-task-load processing nodes and low-task-load processing nodes.

9. The cloud computing-based health big data platform architecture optimization method according to claim 7, characterized in that: The task processing data of each task processing node includes a task execution time fluctuation coefficient, a task failure rate, a resource saturation and an average task execution time of each task processing node within a preset period.

10. A cloud computing-based health big data platform architecture optimization system, characterized in that: include: The data transmission architecture optimization module is used to analyze the data transmission process of the health big data platform, obtain the data transmission optimization information of the health big data platform, and optimize the data transmission architecture configuration of the health big data platform based on the data transmission optimization information of the health big data platform; A data storage partitioning module is used to store the health data after the data transmission optimization configuration is optimized, and divide it into several segments according to time, which are recorded as each health data storage segment; A storage architecture modulation module is used to analyze the query data of each health data storage segment, and then perform storage architecture modulation on the data of each health data storage segment of the health big data platform; The computing resource optimization management module is used to count the task nodes of the health big data platform after the storage architecture is modulated, recorded as task processing nodes, monitor and analyze the task processing process of each task processing node, and then optimize the elastic computing resources of the health big data platform.

Images