Data query method and system for health archives of crowds in super-large area

Abstract

The invention relates to the technical field of big data resource services, and discloses a data query method and system for health archives of crowds in an ultra-large area, and the method comprises the steps: defining an electronic health archive data set, executing a Hash classification strategy according to a fragmentation key field and the electronic health archive data set, and obtaining a health archive database; classifying the electronic health record records into different databases; obtaining a current query condition, executing field correlation analysis, generating a field correlation matrix, and simplifying query; executing a preprocessing strategy, caching the high-frequency non-fragmentation key fields, and performing aggregation processing on the non-fragmentation key fields to obtain non-fragmentation key aggregation fields; according to a historical query data set, executing a multi-target optimization prediction strategy on the aggregation field, and predicting an optimal query path of the non-fragmented key aggregation field; and if the query load is greater than or equal to the load threshold, prompting abnormity. The system can flexibly cope with high-concurrency query, and the query performance and stability are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of big data resource services, and particularly to a data query method and system for health records of a population in an extra-large area. Background Art

[0002] For the management of health records of a population in an extra-large area (including more than one city area or more than one provincial area or a national area), the electronic health record system is usually used to centrally manage the health record information of residents. The electronic health record system contains a large amount of residents' health data, such as multi-source heterogeneous data such as medical records, diagnosis and treatment records, and drug use history. As the multi-source heterogeneous health data increases, a single table or a single database cannot well meet the performance requirements for the addition, deletion, modification, and query of health data, and it is necessary to split the table. Therefore, dynamic aggregation and distributed storage management are used to disperse the multi-source heterogeneous health data into multiple tables or multiple databases.

[0003] There are many problems in the traditional distributed health database management method, especially when facing large-scale health data and high-concurrency queries, the performance problems are particularly prominent. First, the traditional database sharding and table partitioning technology often uses fixed sharding rules (such as time, region, etc.) to split data. This method is difficult to adapt to the dynamically changing health data access patterns, resulting in uneven distribution of health data. Some health database nodes may become query bottlenecks, seriously affecting the performance and stability of the system. Second, the traditional query optimization is usually based on simple rules and does not fully consider the field correlation between query conditions, resulting in unnecessary fields being queried during the query process, increasing the consumption of computing resources and the cost of storage access. In addition, when facing complex query paths, the traditional query optimization methods often do not comprehensively consider multiple objectives such as query time, network transmission cost, storage access cost, and computing resource consumption, and cannot effectively balance between multiple objectives, resulting in low query efficiency of health data and low utilization efficiency of system resources.

[0004] The present invention provides a data query method for health records of a population in an extra-large area to solve the problems raised in the above background art. Summary of the Invention

[0005] The present invention provides a data query method and system for health records of a population in an extra-large area to help solve the problems mentioned in the above background art.

[0006] The present invention provides the following technical solutions: A data query method for health records of a population in an extra-large area, including:

[0007] S1. Define the electronic health record data set D = {d1, d2,..., d n}, where d iDenote the i-th electronic health record. Each electronic health record contains a shard key field and non-shard key fields. n is the number of electronic health records, and i is a positive integer;

[0008] S2. According to the shard key field and the electronic health record dataset, execute a hash classification strategy to classify the electronic health records into different databases;

[0009] The database is used to store the electronic health record dataset;

[0010] S3. Obtain the historical query dataset H = {(q1, p1), (q2, p2), …, (q v , p v )}, where each historical query data includes a query path p t and a query condition q t = (f1, f2…, f k ), f1, f2…, f k are the fields of the query condition, f1, f2…, f k include the shard key field and non-shard key fields, k is the number of fields, and v is the number of historical query data;

[0011] Obtain the current query condition, execute field correlation analysis, generate a field correlation matrix, and simplify the query;

[0012] S4. For the historical query dataset, execute a preprocessing strategy to cache the high-frequency non-shard key fields and perform an aggregation process on the non-shard key fields to obtain a non-shard key aggregation field;

[0013] S5. According to the historical query dataset, execute a multi-objective optimization prediction strategy on the non-shard key aggregation field to predict the optimal query path of the non-shard key aggregation field;

[0014] S6. Execute a query load calculation strategy to obtain the query load;

[0015] Set a load threshold;

[0016] Compare the query load with the load threshold. If the query load is greater than or equal to the load threshold, an exception is prompted.

[0017] Optionally, the step of according to the shard key field and the electronic health record dataset, executing a hash classification strategy to classify the electronic health records into different databases includes:

[0018] Obtain the shard key fields of each electronic health record in the electronic health record dataset to form a shard key field set S = {s1, s2, …, s m}, where s o is the electronic health record do The sharding key field, where 1 ≤ o ≤ m and m is a positive integer;

[0019] Calculate the hash value h(s o );

[0020] Obtain the number a of databases, where a is a positive integer;

[0021] Calculate B j = h(s o ), where B j represents the j-th database and is used to store the electronic health record where h(s o ) mod a = j. The operator mod is the modulo operator.

[0022] Optionally, obtaining the current query condition, performing field correlation analysis, generating a field correlation matrix, and simplifying the query includes:

[0023] Obtain the current query condition q' = (f1, f2…, f k );

[0024] Set the correlation matrix R. The elements in the correlation matrix R represent the correlation measure between field and field ;

[0025] where are the values of field in the l-th electronic health record respectively, and are the means of field respectively;

[0026]

[0027] Set the correlation threshold;

[0028] If is greater than or equal to the correlation threshold, it is determined that field is relevant, merge field , and use the merged field as the first-stage query field;

[0029] If is less than the correlation threshold, it is determined that field is not relevant, then use field as the second-stage query field;

[0030] Use the sharding key field in the current query condition as the first-stage query field;

[0031] When querying the database using the current query conditions, first query the database using the first-stage query fields. When the query result is empty, then query the database using the second-stage query fields.

[0032] Optionally, the multi-objective optimization prediction strategy is performed on the non-sharded key aggregation fields according to the historical query dataset to predict the optimal query path for the non-sharded key aggregation fields, including:

[0033] The multi-objectives include the query execution time C1(p(q')), the network transmission cost C2(p(q')), the storage access cost C3(p(q')), and the computing resource consumption C4(p(q')), where p(q') represents the query path of the current query condition;

[0034] Obtain the multi-objective optimization function:

[0035]

[0036] Use a neural network model to calculate the multi-objective optimization function;

[0037] Use the historical query dataset to train the neural network model, and each query path corresponds to the query execution time, the network transmission cost, the storage access cost, and the computing resource consumption;

[0038] Calculate the loss function:

[0039] where τ(θ) is the loss function, θ generally refers to the weights and biases of the neural network model, and C1(p(q')), C2(p(q')), C3(p(q')), and C4(p(q')) are the true query execution time, network transmission cost, storage access cost, and computing resource consumption respectively;

[0040] and are the query execution time, network transmission cost, storage access cost, and computing resource consumption output by the neural network model respectively;

[0041] and respectively represent the squares of the errors between the true values and the predicted values of the query execution time, network transmission cost, storage access cost, and computing resource consumption.

[0042] Optionally, the multi-objective optimization prediction strategy is performed on the non-sharded key aggregation fields according to the historical query dataset to predict the optimal query path for the non-sharded key aggregation fields, and further includes:

[0043] Optimize the loss function;

[0044] Calculate the partial derivative of the loss function with respect to the parameter θ Among them, τ is the loss function;

[0045] Calculate the updated parameters Among them, θ (t+1) , θ (t) are the neural network parameters for the t-th and (t + 1)-th iterations respectively, η is the learning rate, which controls the step size of each update, is the gradient of the loss function with respect to the parameter θ, is the exponentially weighted moving average of the gradient, ε is a constant used to prevent division by zero, is the normalized gradient;

[0046] Set the error threshold;

[0047] Calculate the difference between two adjacent loss functions and compare the difference with the error threshold;

[0048] If the difference is less than the error threshold, stop optimizing the loss function, obtain the optimized parameters, and obtain the optimal query path based on the optimized parameters.

[0049] Optionally, for the historical query dataset, execute a preprocessing strategy to cache the high-frequency non-sharded key fields and aggregate the non-sharded key fields to obtain non-sharded key aggregation fields, including:

[0050] For each query condition and the corresponding query path in the historical query dataset, mark the query whose result in the query path is cached as a marked query;

[0051] Obtain the total number of marked queries in the historical query dataset, denoted as the cache hit count;

[0052] Calculate the cache hit rate = cache hit count / n;

[0053] Obtain the cache read time;

[0054] Calculate the cache query time = cache read time * cache hit rate;

[0055] Set the query time threshold;

[0056] Compare the cache query time with the query time threshold. If the cache query time is less than or equal to the query time threshold, it is determined that the non-sharded key field in the query condition is a high-frequency non-sharded key field.

[0057] Optionally, for the historical query dataset, execute a preprocessing strategy to cache the high-frequency non-sharded key fields and aggregate the non-sharded key fields to obtain non-sharded key aggregation fields, further including:

[0058] Obtain all high-frequency non-sharded key fields and cache the high-frequency non-sharded key fields;

[0059] Arbitrarily obtain two non-sharded key fields f from the historical query dataset e1 , f e2 ;

[0060] Calculate the non-linear relationship of f e1 , f e2 where P(x , x e1 , x e2 ) is the joint probability distribution when the non-sharded key field f e1 , f e2 takes the value of x e1 , x e2 , P(x e1 ), P(x e2 ) distributions represent the marginal probability distributions of the non-sharded key field f e1 , f e2 . Among them, Calculate the logarithm of the ratio between the joint probability distribution and the marginal probability distribution to quantify the non-linear relationship between fields f e1 , f e2 . Multiply by the joint probability distribution in to weight the contribution of each logarithmic term. The meaning of accumulation is to calculate the corresponding terms for all pairs of x e1 , x e2 values, used to measure the non-linear relationship between two fields f e1 , f e2 ;

[0061] Compare the magnitude relationship between I(f e1 , f e2 ) and 0. If I(f e1 , f e2 ) ≥ 0, then aggregate the non-sharded key fields f e1 , f e2 to obtain a non-sharded key aggregated field and store it in the database;

[0062] If I(f e1 , f e2 ) < 0, then it is determined that the non-sharded key fields f e1 , f e2 are independent of each other;

[0063] Obtain the current query conditions;

[0064] First, use cache query to obtain the high-frequency non-sharded key fields in the cache, compare them with the fields in the query conditions, and determine whether there are high-frequency non-sharded key fields in the query conditions;

[0065] If so, return the cache result;

[0066] If not, perform a database query, specifically:

[0067] Obtain the non-sharding key aggregation fields in the database, compare them with the fields in the query condition, and determine whether the fields in the query condition contain non-sharding key aggregation fields;

[0068] If it contains non-sharding key aggregation fields, return the results corresponding to the non-sharding key aggregation fields;

[0069] If it does not contain non-sharding key aggregation fields, sequentially obtain the fields in the query condition and query the database.

[0070] Optionally, the execution of the query load calculation strategy to obtain the query load includes:

[0071] Obtain the complexity U(q z ), 1 ≦ z ≦ w, where w is the number of current queries;

[0072] Calculate the query load where q z represents the z-th current query, Accumulate the complexities of the current w queries as the query load.

[0073] A data query system for the health records of a large population in a large area, including:

[0074] Electronic health record dataset management module: Manage electronic health record data and partition the data according to the sharding key field;

[0075] Hash classification strategy module: Allocate data to different databases according to the sharding key field through the hash algorithm;

[0076] Query processing and optimization module: Simplify the query through field correlation analysis, query path prediction, and multi-objective optimization;

[0077] Nonlinear relationship analysis and field aggregation module: Analyze the relationships between high-frequency non-sharding key fields, judge relevant fields through joint probability distribution, and perform aggregation optimization queries;

[0078] High-frequency non-sharding key field caching and aggregation module: Cache high-frequency query fields and perform aggregation;

[0079] Neural network training and multi-objective optimization module: Use historical query data to train a neural network model, optimize the query path, and balance multiple objectives;

[0080] Query load monitoring and warning module: Calculate the query load in real time, monitor the system performance, and issue a warning when the load exceeds the set threshold.

[0081] The present invention has the following beneficial effects:

[0082] 1. For the data query method of the health records of a large population in a large area, through real-time monitoring and anomaly detection of the query load, this solution can effectively evaluate the system pressure and make a quick response. When the query load exceeds the preset threshold, the system will promptly identify and trigger an anomaly warning. This provides an early warning for maintenance personnel to avoid performance degradation or system crashes when the load is too high. By setting the load threshold, the system can dynamically adjust the strategy according to the actual load situation, ensuring both fast data query and avoiding service interruption caused by excessive query pressure. By adopting this load monitoring mechanism, the system can flexibly handle high-concurrency queries and take measures in case of anomalies, such as warning, resource expansion, flow limiting, etc., thus ensuring query performance and system stability.

[0083] 2. For the data query method of the health records of a large population in a large area, by adopting a hash classification strategy to fragment the electronic health record records, the system can efficiently distribute the data into multiple databases to optimize the query response time and resource utilization. The hash classification strategy ensures the uniform distribution of data among different databases by calculating the hash values of the fragment key fields. This strategy can effectively avoid excessive load on some databases caused by uneven data distribution, thereby improving the parallelism of the system during query and reducing resource waste. By evenly distributing the records of each fragment into different databases, the system can achieve efficient data storage and access, and ensure that the load of each database is within a certain range to avoid performance bottlenecks. Through the hash algorithm, the system can disperse the data to different storage nodes based on the fragment key fields, reducing the pressure on some nodes. For a large amount of electronic health record data, hash classification not only improves the query efficiency but also optimizes the storage resources, with high scalability to adapt to the future growth of data volume.

[0084] 3. For the data query method of the health records of a large population in a large area, performing field correlation analysis can further optimize the query path and reduce the calculation cost. By analyzing the correlation between query conditions, the system can identify which fields are highly correlated, and then merge the relevant fields, thus simplifying the query operation. For example, if multiple fields in a query are highly correlated, the system can merge the query operations to avoid scanning the data multiple times and improve the query efficiency. By reducing the redundant operations in the query, the order and manner of data access are optimized, avoiding unnecessary calculation overhead.

[0085] 4. The data query method for the health records of the population in an extremely large area. By implementing a multi-objective optimization prediction strategy, the system can achieve a balance among multiple optimization objectives and finally predict the optimal query path. This strategy takes into account multiple objectives such as query execution time, network transmission cost, storage access cost, and computing resource consumption, ensuring that the query can achieve the optimal balance in these dimensions. Using a neural network model for multi-objective optimization enables the system to learn based on historical query datasets and be more intelligent and accurate when predicting query paths. By training the neural network model, the system can combine the prediction model to output the optimal query path during actual queries, minimizing resource waste to the greatest extent. Through multi-objective optimization, the performance of the query system can be comprehensively improved, and the best query solution can be provided under changing query requirements, thus ensuring the efficient and stable operation of the system.

[0086] 5. The data query method for the health records of the population in an extremely large area. In multi-objective optimization, optimizing the loss function is the core step in training the neural network model. By calculating the loss function and performing backpropagation, the neural network can adjust its weights to make the predicted values closer to the actual values, thereby improving the accuracy of query path prediction. In multi-objective optimization, the loss function takes into account not only query execution time but also multiple dimensions such as network transmission cost, storage access cost, and computing resource consumption. During optimization, the system calculates the partial derivatives of the loss function with respect to the parameters and gradually adjusts the network model until the error threshold is met. This enables the system to improve the accuracy of query path prediction through continuous learning and optimization. The optimization process of the loss function ensures the balance among various objectives and avoids the unbalanced utilization of system resources caused by over-optimization of a single objective.

[0087] 6. The data query method for the health records of the population in an extremely large area. For non-sharding key fields with high-frequency queries, this solution can effectively improve the query response speed through caching and aggregation processing strategies. By storing the query results of high-frequency fields in the cache, the system can avoid repeated queries to the database and reduce the query pressure on the database. In addition, the system will regularly perform aggregation processing on high-frequency fields, pre-calculate and store the results, reducing the computational amount during real-time queries. The caching mechanism can further reduce query time by increasing the cache hit rate and optimizing query performance. Through the caching and aggregation mechanisms, the access frequency to the database is effectively reduced, the pressure on the database is alleviated, and the performance bottleneck caused by high-concurrency queries to the system is avoided.

[0088] 7. The data query method for the health records of a large population in a large area can further optimize the data aggregation strategy by calculating the non-linear relationships between high-frequency non-sharded key fields. By identifying which high-frequency fields have significant correlations, the system can merge relevant fields during querying, reducing the number of queries and data processing time. Through the calculation of joint probability distributions, the system can automatically evaluate the non-linear relationships between fields, thereby determining which fields can be merged and which fields need to be queried separately. Automatically adjusting the query strategy according to the relevance of the actual data avoids redundant query operations. By deeply analyzing the non-linear relationships between fields, the system can make more intelligent and accurate decisions, thus improving the overall performance of the query and the resource utilization efficiency.

[0089] 8. The data query method for the health records of a large population in a large area can, through the calculation of query load, enable the system to evaluate the current query pressure in real time and dynamically adjust the strategy according to the query complexity. The calculation of query load takes into account the complexity of each query, helping the system determine whether load balancing, flow limiting and other measures need to be taken currently. By obtaining the complexity of each query, the system can accurately evaluate the load situation and make appropriate responses. It provides a tool for quantifying query load, enabling the system to monitor the query pressure in real time and dynamically adjust according to the query complexity. By accurately calculating the query load, the system can avoid overload or performance bottlenecks and improve the stability and response speed of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a schematic flow diagram of the method of the present invention.

[0091] Figure 2 It is a schematic diagram of the system modules of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0092] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0093] Embodiment 1, referring to Figure 1 , a data query method for the health records of a large population in a large area, includes:

[0094] S1. Define the electronic health record data set D = {d1, d2,..., d n}, where d iDenote the i-th electronic health record. Each electronic health record contains a shard key field and a non-shard key field. n is the number of electronic health records, and i is a positive integer;

[0095] In this embodiment, the shard key field is the field that determines the distribution of electronic health records, directly participates in the sharding rule, and is usually a frequently queried or primary key field. The non-shard key field is other fields except the shard key field, mainly used to store business data, and does not affect the sharding distribution of data.

[0096] S2. According to the shard key field and the electronic health record dataset, execute the hash classification strategy to classify the electronic health records into different databases;

[0097] The database is used to store the electronic health record dataset;

[0098] S3. Obtain the historical query dataset H = {(q1, p1), (q2, p2), …, (q v , p v )}, where each historical query data includes a query path p t and a query condition q t = (f1, f2…, f k ), f1, f2…, f k are the fields of the query condition, f1, f2…, f k include the shard key field and the non-shard key field, k is the number of fields, and v is the number of historical query data;

[0099] Obtain the current query condition, execute the field correlation analysis, generate a field correlation matrix, and simplify the query;

[0100] S4. For the historical query dataset, execute the preprocessing strategy to cache the frequently-occurring non-shard key fields and perform aggregation processing on the non-shard key fields to obtain non-shard key aggregation fields;

[0101] S5. According to the historical query dataset, execute the multi-objective optimization prediction strategy on the non-shard key aggregation fields to predict the optimal query path of the non-shard key aggregation fields;

[0102] S6. Execute the query load calculation strategy to obtain the query load;

[0103] Set the load threshold;

[0104] Compare the query load with the load threshold. If the query load is greater than or equal to the load threshold, an exception is prompted.

[0105] Through real-time monitoring and anomaly detection of the query load, this solution can effectively evaluate the system pressure and make a quick response. When the query load exceeds the preset threshold, the system will promptly identify and trigger an anomaly warning. This provides an early alert for maintenance personnel and avoids performance degradation or system crashes when the load is too high. By setting the load threshold, the system can dynamically adjust the strategy according to the actual load situation, ensuring both fast query of data and avoiding service interruptions caused by excessive query pressure. With this load monitoring mechanism, the system can flexibly handle high-concurrency queries and take measures in case of anomalies, such as warning, resource expansion, flow limiting, etc., thus ensuring query performance and system stability.

[0106] The hash classification strategy is executed according to the shard key field and the electronic health record dataset to classify the electronic health record into different databases, including:

[0107] Obtain the shard key fields of each electronic health record in the electronic health record dataset to form a shard key field set S = {s1, s2, …, s m}, where s o is the shard key field of the electronic health record d o , where 1 ≤ o ≤ m and m is a positive integer;

[0108] Calculate the hash value h(s o );

[0109] Obtain the number of databases a, where a is a positive integer;

[0110] Calculate B j = h(s o ) mod a, where B j represents the jth database and is used to store the electronic health record for which h(s o ) mod a = j, and the operator mod is the modulo operator.

[0111] By adopting a hash classification strategy to shard electronic health record entries, the system can efficiently distribute data across multiple databases to optimize query response time and resource utilization. The hash classification strategy ensures uniform distribution of data among different databases by calculating the hash values of shard key fields. This strategy can effectively avoid excessive load on certain databases caused by uneven data distribution, thereby enhancing the parallelism of the system during querying and reducing resource waste. By evenly distributing the records of each shard to different databases, the system can achieve efficient data storage and access, and ensure that the load on each database is within a certain range to avoid performance bottlenecks. Through the hash algorithm, the system can disperse data to different storage nodes based on shard key fields, alleviating the pressure on certain nodes. For massive electronic health record data, hash classification not only improves query efficiency but also optimizes storage resources, with high scalability to accommodate future data volume growth.

[0112] The obtaining of the current query conditions, performing field correlation analysis, and generating a field correlation matrix to simplify the query includes:

[0113] Obtain the current query conditions \(q'=(f_1,f_2,\cdots,f\) k );

[0114] Set the correlation matrix \(R\), and the elements in the correlation matrix \(R\) represent the correlation metric between field and field ;

[0115] where are respectively the values of field in the \(l\)-th electronic health record entry, are respectively the means of field ;

[0116]

[0117] Set the correlation threshold;

[0118] If is greater than or equal to the correlation threshold, it is determined that field is relevant, merge field , and use the merged field as the query field in the first stage;

[0119] If is less than the correlation threshold, it is determined that field is not relevant, then use field as the query field in the second stage;

[0120] Use the shard key field in the current query condition as the first-stage query field;

[0121] When querying the database using the current query condition, first query the database using the first-stage query field. When the query result is empty, then query the database using the second-stage query field.

[0122] Performing field correlation analysis can further optimize the query path and reduce the computational cost. By analyzing the correlation between query conditions, the system can identify which fields are highly correlated, and then merge the related fields, thus simplifying the query operation. For example, if multiple fields in the query are highly correlated, the system can merge the query operations, avoid scanning the data multiple times, and improve the query efficiency. The generation of the correlation matrix is automated and does not require manual intervention. It can be dynamically adjusted during each query to ensure the optimal combination of query conditions. By reducing redundant operations in the query, the order and manner of data access are optimized, and unnecessary computational overhead is avoided.

[0123] Performing a multi-objective optimization prediction strategy on the non-shard key aggregation field according to the historical query dataset to predict the optimal query path for the non-shard key aggregation field, including:

[0124] The multi-objectives include query execution time C1(p(q')), network transmission cost C2(p(q')), storage access cost C3(p(q')), and computational resource consumption C4(p(q')), where p(q') represents the query path of the current query condition;

[0125] Obtain the multi-objective optimization function:

[0126]

[0127] Use a neural network model to calculate the multi-objective optimization function;

[0128] Use the historical query dataset to train the neural network model, and each query path corresponds to query execution time, network transmission cost, storage access cost, and computational resource consumption;

[0129] Calculate the loss function:

[0130]

[0131] where τ(θ) is the loss function, θ generally refers to the weights and biases of the neural network model, and C1(p(q')), C2(p(q')), C3(p(q')), and C4(p(q')) are the actual query execution time, network transmission cost, storage access cost, and computational resource consumption respectively;

[0132] and They are the query execution time, network transmission cost, storage access cost, and computing resource consumption output by the neural network model respectively;

[0133] and respectively represent the squares of the errors between the true values and the predicted values of the query execution time, network transmission cost, storage access cost, and computing resource consumption.

[0134] In this embodiment, the existing neural network model can be any one of a feedforward neural network, a convolutional neural network, a recurrent neural network, or a long short-term memory network.

[0135] By executing the multi-objective optimization prediction strategy, the system can achieve a balance among multiple optimization objectives and finally predict the optimal query path. This strategy takes into account multiple objectives such as query execution time, network transmission cost, storage access cost, and computing resource consumption, ensuring that the query can achieve an optimal balance in these dimensions. Using a neural network for multi-objective optimization enables the system to learn based on historical query datasets and be more intelligent and accurate when predicting query paths. By training the neural network model, the system can combine the prediction model to output the optimal query path during actual queries, minimizing resource waste to the greatest extent. Through multi-objective optimization, the performance of the query system can be comprehensively improved, and the best query plan can be provided under changing query requirements, thus ensuring the efficient and stable operation of the system.

[0136] The step of performing a multi-objective optimization prediction strategy on the non-sharded key aggregation field according to the historical query dataset to predict the optimal query path of the non-sharded key aggregation field further includes:

[0137] Optimizing the loss function;

[0138] Calculating the partial derivative of the loss function with respect to the parameter θ where τ is the loss function;

[0139] Calculating the updated parameter where θ (t+1) , θ (t) are the neural network parameters of the t-th and (t + 1)-th iterations respectively, η is the learning rate, controlling the step size of each update, is the gradient of the loss function with respect to the parameter θ, is the exponentially weighted mean of the gradient, ε is a constant used to prevent division by zero, is the normalized gradient;

[0140] Setting an error threshold;

[0141] Calculating the difference between two adjacent loss functions and comparing the difference with the error threshold;

[0142] If the difference is less than the error threshold, stop optimizing the loss function, obtain the optimized parameters, and obtain the optimal query path based on the optimized parameters.

[0143] In multi-objective optimization, optimizing the loss function is a core step in training a neural network model. By calculating the loss function and performing backpropagation, the neural network can adjust the weights to make the predicted values closer to the actual values, thereby improving the accuracy of query path prediction. In multi-objective optimization, the loss function considers not only the query execution time but also multiple dimensions such as network transmission cost, storage access cost, and computing resource consumption. During optimization, the system calculates the partial derivatives of the loss function with respect to the parameters and gradually adjusts the network model until the error threshold is met. This enables the system to improve the accuracy of query path prediction through continuous learning and optimization. The optimization process of the loss function ensures the balance between various objectives and avoids the unbalanced utilization of system resources due to excessive optimization of a single objective.

[0144] For the historical query dataset, execute a preprocessing strategy to cache the high-frequency non-sharded key fields and aggregate the non-sharded key fields to obtain non-sharded key aggregation fields, including:

[0145] For each query condition and the corresponding query path in the historical query dataset, mark the query whose result in the cache as a marked query;

[0146] Obtain the total number of marked queries in the historical query dataset, denoted as the cache hit count;

[0147] Calculate the cache hit rate = cache hit count / n;

[0148] Obtain the cache read time;

[0149] Calculate the cache query time = cache read time * cache hit rate;

[0150] Set the query time threshold;

[0151] Compare the cache query time with the query time threshold. If the cache query time is less than or equal to the query time threshold, it is determined that the non-sharded key field in the query condition is a high-frequency non-sharded key field.

[0152] For non-sharded key fields with high-frequency queries, this solution can significantly improve query response speed through caching and aggregation strategies. By storing the query results of high-frequency fields in the cache, the system can avoid repeated queries to the database and reduce the query pressure on the database. In addition, the system will regularly perform aggregation processing on high-frequency fields, pre-compute and store the results, and reduce the amount of computation during real-time queries. The caching mechanism can further reduce query time by increasing the cache hit rate and optimize query performance. Through the caching and aggregation mechanisms, the access frequency to the database is significantly reduced, the pressure on the database is alleviated, and the performance bottleneck caused by high-concurrency queries in the system is avoided.

[0153] For the historical query dataset, execute a preprocessing strategy to cache high-frequency non-sharded key fields and perform aggregation processing on non-sharded key fields to obtain non-sharded key aggregation fields, and further include:

[0154] Obtain all high-frequency non-sharded key fields and cache the high-frequency non-sharded key fields;

[0155] Arbitrarily obtain two non-sharded key fields f e1 , f e2 ;

[0156] Calculate the non-linear relationship of f e1 , f e2 ; where P(x e1 , x e2 ) is the joint probability distribution of non-sharded key fields f e1 , f e2 taking values x e1 , x e2 , P(x e1 ), P(x e2 ) distributions represent the marginal probability distributions of non-sharded key fields f e1 , f e2 , and among them, Calculate the logarithm of the ratio between the joint probability distribution and the marginal probability distribution to quantify the non-linear relationship between fields f e1 , f e2 ; Multiply by the joint probability distribution in e1 , x e2 to weight the contribution of each logarithmic term, and the meaning of accumulation is to calculate the corresponding terms for all pairs of values of x for measuring the non-linear relationship between two fields f e1 , f e2 ;

[0157] Compare I(f e1 , f e2) and the size relationship with 0. If I(f e1 , f e2 ) ≥ 0, then aggregate the non-sharding key fields f e1 , f e2 to obtain the non-sharding key aggregation field and store it in the database;

[0158] If I(f e1 , f e2 ) < 0, then it is determined that the non-sharding key fields f e1 , f e2 are independent of each other;

[0159] Obtain the query conditions for this time;

[0160] First, use cache query to obtain the high-frequency non-sharding key fields in the cache, compare them with the fields in the query conditions, and determine whether there are high-frequency non-sharding key fields in the query conditions;

[0161] If there are, return the cache result;

[0162] If not, perform a database query. Specifically:

[0163] Obtain the non-sharding key aggregation fields in the database, compare them with the fields in the query conditions, and determine whether the fields in the query conditions contain non-sharding key aggregation fields;

[0164] If it contains non-sharding key aggregation fields, return the results corresponding to the non-sharding key aggregation fields;

[0165] If it does not contain non-sharding key aggregation fields, sequentially obtain the fields in the query conditions and query the database.

[0166] By calculating the non-linear relationship between high-frequency non-sharding key fields, the system can further optimize the data aggregation strategy. By identifying which high-frequency fields have significant correlations, the system can merge related fields during query, reducing the number of queries and data processing time. Through the calculation of the joint probability distribution, the system can automatically evaluate the non-linear relationship between fields, thereby determining which fields can be merged and which fields need to be queried separately. By deeply analyzing the non-linear relationship between fields, the system can make more intelligent and accurate decisions, thus improving the overall performance of the query and the resource utilization efficiency.

[0167] The execution of the query load calculation strategy to obtain the query load includes:

[0168] Obtain the complexity U(q z ) of each current query, where 1 ≤ z ≤ w, and w is the number of current queries;

[0169] Calculate the query load where qz Represents the current z-th query, Accumulate the complexity of the current w queries as the query load.

[0170] Through the calculation of the query load, the system can evaluate the pressure of the current query in real time and dynamically adjust the strategy according to the query complexity. The calculation of the query load takes into account the complexity of each query, helping the system determine whether measures such as load balancing and flow limiting need to be taken currently. By obtaining the complexity of each query, the system can accurately evaluate the load situation and make appropriate responses. It provides a tool for quantifying the query load, enabling the system to monitor the query pressure in real time and make dynamic adjustments according to the query complexity. By accurately calculating the query load, the system can avoid overload or performance bottlenecks and improve the stability and response speed of the system.

[0171] Embodiment 2, referring to Figure 2 , A data query system for health records of a large population in a large area, including:

[0172] Electronic health record data set management module: Manage electronic health record data and partition the data according to the sharding key field;

[0173] Hash classification strategy module: Allocate data to different databases according to the sharding key field through the hash algorithm;

[0174] Query processing and optimization module: Simplify the query through field relevance analysis, query path prediction, and multi-objective optimization;

[0175] Nonlinear relationship analysis and field aggregation module: Analyze the relationships between high-frequency non-sharding key fields, judge relevant fields through joint probability distribution, and perform aggregation to optimize the query;

[0176] High-frequency non-sharding key field caching and aggregation module: Cache high-frequency query fields and perform aggregation;

[0177] Neural network training and multi-objective optimization module: Use historical query data to train a neural network model, optimize the query path, and balance multiple objectives;

[0178] Query load monitoring and warning module: Calculate the query load in real time, monitor the system performance, and issue a warning when the load exceeds the set threshold.

[0179] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0180] The above are only the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for querying data of a health record of a population in an extremely large area, characterized in that: Including: S1. Define the electronic health record dataset D = {d1, d2, …, d n}, where d i represents the i-th electronic health record. Each electronic health record contains a sharding key field and a non-sharding key field. n is the number of electronic health records, and i is a positive integer; S2. Execute a hash classification strategy based on the sharding key field and the electronic health record dataset, and classify the electronic health record records into different databases; The databases are used to store the electronic health record dataset; S3. Obtain the historical query dataset H = {(q1, p1), (q2, p2), …, (q v , p v )}, where each historical query data includes a query path p t and a query condition q t = (f1, f2, …, f k ), f1, f2, …, f k are the fields of the query condition, f1, f2, …, f k include a shard key field and non-shard key fields, k is the number of fields, v is the number of historical query data, 1 ≤ t ≤ v, and v is a positive integer; Obtain the current query conditions, perform field correlation analysis, generate a field correlation matrix, and simplify the query; S4. For the historical query dataset, execute a preprocessing strategy, cache the high-frequency non-sharding key fields, and perform aggregation processing on the non-sharding key fields to obtain non-sharding key aggregation fields; S5. According to the historical query dataset, perform a multi-objective optimization prediction strategy on the non-sharding key aggregation fields to predict the optimal query path of the non-sharding key aggregation fields; S6. Execute a query load calculation strategy to obtain the query load; Set a load threshold; Compare the query load with the load threshold. If the query load is greater than or equal to the load threshold, an exception is prompted.

2. The data query method for the health records of a large population in a large area according to claim 1, wherein: The step of executing a hash classification strategy based on the sharding key field and the electronic health record dataset to classify the electronic health record records into different databases includes: Obtain the shard key fields of each electronic health record in the electronic health record dataset to form a shard key field set S = {s1, s2, …, s m}, where s o is the shard key field of the electronic health record d o , where 1 ≤ o ≤ m and m is a positive integer; Compute the hash value h(s o ) of the sharding key field; Obtain the number a of databases, where a is a positive integer; Calculate B j = h(s o ) mod a, where B j represents the j-th database for storing electronic health record records where h(s o ) mod a = j, and the operator mod is the modulo operator.

3. The data query method for the health records of a large population in an extremely large area according to claim 1, wherein: The step of obtaining the current query conditions, performing field correlation analysis, generating a field correlation matrix, and simplifying the query includes: Obtain the query condition for this time q'=(f1,f2…,f k ); Set the correlation matrix R, where the elements in the correlation matrix R represent the field and the field the correlation measure between them; wherein, are respectively the values of the fields in the l-th electronic health record, are respectively the means of the fields ; Set a correlation threshold; If is greater than or equal to the relevance threshold, then it is determined that the field is relevant, and the fields are merged, and the merged field is used as the query field for the first stage; If is less than the relevance threshold, it is determined that the field is not relevant, and the field is used as the query field in the second stage; Take the sharding key field in the current query conditions as the first-stage query field; When querying the database using the current query conditions, first query the database using the first-stage query field. When the query result is empty, then query the database using the second-stage query field.

4. The data query method for the health records of a super-large area population according to claim 3, characterized in that: The step of performing a multi-objective optimization prediction strategy on the non-sharding key aggregation fields according to the historical query dataset to predict the optimal query path of the non-sharding key aggregation fields includes: The multi-objectives include query execution time C1(p(q')), network transmission cost C2(p(q')), storage access cost C3(p(q')), and computing resource consumption C4(p(q')), where p(q') represents the query path of the current query conditions; Obtain a multi-objective optimization function: Use a neural network model to calculate the multi-objective optimization function; Use the historical query dataset to train the neural network model, and each query path corresponds to query execution time, network transmission cost, storage access cost, and computing resource consumption; Calculate the loss function: Among them, τ(θ) is the loss function, θ generally refers to the weights and biases of the neural network model, and C1(p(q')), C2(p(q')), C3(p(q')), and C4(p(q')) are the true query execution time, network transmission cost, storage access cost, and computing resource consumption respectively; and are the query execution time, network transmission cost, storage access cost, and computing resource consumption output by the neural network model, respectively; and respectively represent the squares of the errors between the true values and the predicted values of the query execution time, network transmission cost, storage access cost, and computing resource consumption.

5. The data query method for the ultra-large area population health records according to claim 4, characterized in that: The step of performing a multi-objective optimization prediction strategy on the non-sharding key aggregation fields according to the historical query dataset to predict the optimal query path of the non-sharding key aggregation fields further includes: Optimize the loss function; Calculate the partial derivative of the loss function with respect to the parameter θ where τ is the loss function; Calculate the updated parameters where θ (t+1) , θ (t) are the neural network parameters for the t-th and (t + 1)-th iterations respectively, η is the learning rate that controls the step size of each update, is the gradient of the loss function with respect to the parameter θ, is the exponentially weighted mean of the gradient, and ε is a constant used to prevent division by zero, is the normalized gradient; Set an error threshold; Calculate the difference between two adjacent loss functions, and compare the difference with the error threshold; If the difference is less than the error threshold, stop optimizing the loss function, obtain the optimized parameters, and obtain the optimal query path based on the optimized parameters.

6. The data query method for the ultra-large area population health records according to claim 1, characterized in that: For the historical query dataset, execute a preprocessing strategy to cache high-frequency non-sharded key fields and aggregate the non-sharded key fields to obtain non-sharded key aggregate fields, including: For each query condition and corresponding query path in the historical query dataset, mark the query whose result in the cache is the query path as a marked query; Obtain the total number of marked queries in the historical query dataset, denoted as the cache hit count; Calculate the cache hit rate = cache hit count / n; Obtain the cache read time; Calculate the cache query time = cache read time * cache hit rate; Set a query time threshold; Compare the cache query time with the query time threshold. If the cache query time is less than or equal to the query time threshold, then determine that the non-sharded key field in the query condition is a high-frequency non-sharded key field.

7. The data query method for the health records of a large population in an oversized area according to claim 6, characterized in that: For the historical query dataset, execute a preprocessing strategy to cache high-frequency non-sharded key fields and aggregate the non-sharded key fields to obtain non-sharded key aggregate fields, further including: Obtain all high-frequency non-sharded key fields and cache the high-frequency non-sharded key fields; Arbitrarily obtain two non-sharding key fields f in the historical query dataset e1 , f e2 ; Calculate f e1 , f e2 's non - linear relationship where P(x e1 , x e2 ) is the joint probability distribution of the non - shard key field f e1 , f e2 taking the value of x e1 , x e2 , and P(x e1 ), P(x e2 ) distributions represent the marginal probability distributions of the non - shard key field f e1 , f e2 . Among them, calculate the logarithm of the ratio between the joint probability distribution and the marginal probability distribution, used to quantify the non - linear relationship between fields f e1 , f e2 ; multiplying the joint probability distribution is used to weight the contribution of each logarithmic term, and the meaning of accumulation is to calculate the corresponding terms for all pairs of values of x e1 , x e2 ; used to measure the non - linear relationship between two fields f e1 , f e2 ; Compare I(f e1 ,f e2 ) with 0. If I(f e1 ,f e2 ) ≥ 0, then aggregate the non-sharded key fields f e1 ,f e2 to obtain a non-sharded key aggregation field and store it in the database; If I(f e1 , f e2 ) < 0, it is determined that the non-shard key fields f e1 , f e2 are independent of each other; Obtain the current query condition; First, use cache query to obtain the high-frequency non-sharded key fields in the cache and compare them with the fields in the query condition to determine whether there are high-frequency non-sharded key fields in the query condition; If there are, return the cache result; If not, perform a database query. Specifically: Obtain the non-sharded key aggregate fields in the database and compare them with the fields in the query condition to determine whether the fields in the query condition contain non-sharded key aggregate fields; If they contain non-sharded key aggregate fields, return the result corresponding to the non-sharded key aggregate fields; If they do not contain non-sharded key aggregate fields, sequentially obtain the fields in the query condition and query the database.

8. The data query method for the ultra-large area population health record according to claim 1, wherein: Execute the query load calculation strategy to obtain the query load, including: Obtain the complexity U(q of each current query z ), 1 ≦ z ≦ w, where w is the number of current queries; Calculate query load where q z represents the z-th current query, accumulate the complexity of the current w queries as the query load.

9. A data query system for the health records of a large population in a large area, which is applied to the data query method for the health records of a large population in a large area described in any one of claims 1-8, and is characterized in that, Including: Electronic health record dataset management module: Manage electronic health record data and partition the data according to sharded key fields; Hash classification strategy module: Allocate data to different databases according to sharded key fields through a hash algorithm; Query processing and optimization module: Simplify the query through field correlation analysis, query path prediction, and multi-objective optimization; Nonlinear relationship analysis and field aggregation module: Analyze the relationships between non-sharded key fields and perform aggregation optimization queries; High-frequency non-sharded key field caching and aggregation module: Cache high-frequency query fields and perform aggregation; Neural network training and multi-objective optimization module: Use historical query data to train a neural network model, optimize the query path, and balance multiple objectives; Query load monitoring and warning module: Calculate the query load in real time, monitor the system performance, and issue a warning when the load exceeds the set threshold.

Images