A method and system for querying health records of a large population area

Abstract

This invention relates to the field of big data resource service technology and discloses a data query method and system for health records of a large-scale population. The method includes: defining an electronic health record dataset; performing a hash classification strategy based on the sharding key field and the electronic health record dataset to classify electronic health record records into different databases; obtaining the current query conditions; performing field correlation analysis to generate a field correlation matrix and simplify the query; performing a preprocessing strategy to cache high-frequency non-sharding key fields and aggregate the non-sharding key fields to obtain aggregated non-sharding key fields; performing a multi-objective optimization prediction strategy on the aggregated fields based on historical query datasets to predict the optimal query path for the aggregated non-sharding key fields; and indicating an anomaly if the query load is greater than or equal to a load threshold. The system can flexibly handle high-concurrency queries, ensuring query performance and stability.

Description

Technical Field

[0001] This invention relates to the field of big data resource service technology, specifically to a data query method and system for health records of a population in a very large area. Background Technology

[0002] For the management of health records for populations in very large areas (including more than one prefecture-level city, one or more provincial-level regions, or one national region), electronic health record systems are typically used to centrally manage residents' health record information. Electronic health record systems contain a large amount of residents' health data, such as medical records, treatment records, and medication history, which are multi-source heterogeneous data. As the amount of multi-source heterogeneous health data increases, a single table or a single database cannot adequately meet the performance requirements for adding, deleting, modifying, and querying health data. Therefore, it is necessary to split the tables. Thus, dynamic aggregation and distributed storage management are adopted to distribute multi-source heterogeneous health data into multiple tables or multiple databases.

[0003] Traditional distributed health database management methods suffer from numerous problems, particularly performance issues when dealing with large-scale health data and high-concurrency queries. First, traditional database sharding techniques often employ fixed sharding rules (such as time or region) for data partitioning. This approach struggles to adapt to dynamically changing health data access patterns, leading to uneven data distribution. Some health database nodes may become query bottlenecks, severely impacting system performance and stability. Second, traditional query optimization is typically based on simple rules, failing to adequately consider the correlation between fields in query conditions. This results in unnecessary fields being queried, increasing computational resource consumption and storage access costs. Furthermore, when faced with complex query paths, traditional query optimization methods often fail to comprehensively consider multiple objectives, including query time, network transmission costs, storage access costs, and computational resource consumption. This inability to effectively balance these objectives results in low health data query efficiency and low system resource utilization.

[0004] This invention provides a method for querying health records of a large population area, thus solving the problems mentioned in the background section. Summary of the Invention

[0005] This invention provides a data query method and system for health records of a large population area, which helps to solve the problems mentioned in the background art.

[0006] This invention provides the following technical solution: a method for querying health records of a population in a very large area, comprising:

[0007] S1. Define the electronic health record dataset D = {d1, d2, ..., d...} n}, where d iThis represents the i-th electronic health record. Each electronic health record contains a fragment key field and a non-fragment key field. n is the number of electronic health record records, and i is a positive integer.

[0008] S2. Based on the sharding key field and the electronic health record dataset, execute a hash classification strategy to classify the electronic health record records into different databases;

[0009] The database is used to store electronic health record datasets;

[0010] S3. Obtain the historical query dataset H = {(q1,p1),(q2,p2),…,(q v ,p v )}, where each historical query data includes the query path p t and query condition q t =(f1,f2,…,f k ), f1, f2…, f k These are the fields used in the query conditions, f1, f2, ..., f k Includes sharding key fields and non-sharding key fields, k is the number of fields, and v is the number of historical query data;

[0011] Obtain the query conditions, perform field relevance analysis, generate a field relevance matrix, and simplify the query;

[0012] S4. For the historical query dataset, execute a preprocessing strategy to cache the high-frequency non-sharding key fields and aggregate the non-sharding key fields to obtain the non-sharding key aggregate fields.

[0013] S5. Based on the historical query dataset, perform a multi-objective optimization prediction strategy on the non-sharding key aggregation field to predict the optimal query path for the non-sharding key aggregation field.

[0014] S6. Execute the query load calculation strategy to obtain the query load;

[0015] Set a load threshold;

[0016] The query load is compared with the load threshold. If the query load is greater than or equal to the load threshold, an exception is reported.

[0017] Optionally, the step of performing a hash classification strategy based on the sharding key field and the electronic health record dataset to classify electronic health record records into different databases includes:

[0018] The sharding key field set S = {s1, s2, ..., s...} is composed of the sharding key field of each electronic health record in the electronic health record dataset. m}, where s o For electronic health record recording do The sharding key field, where 1 ≤ o ≤ m, and m is a positive integer;

[0019] Calculate the hash value h(s) of the shard key field. o );

[0020] Get the number of databases, 'a', where 'a' is a positive integer;

[0021] Calculate B j =h(s o )moda, where B j This represents the j-th database, used to store h(s) o The electronic health record record of moda = j, where mod is the modulo operator.

[0022] Optionally, obtaining the current query conditions, performing field relevance analysis, generating a field relevance matrix, and simplifying the query include:

[0023] Get the query conditions q' = (f1, f2, ..., f k );

[0024] Let the correlation matrix R be defined, and the elements in the correlation matrix R be defined. Representation field and fields A measure of the correlation between them;

[0025] in, fields respectively The value in the l-th electronic health record record, fields respectively The mean;

[0026]

[0027] Set a correlation threshold;

[0028] like If the relevance threshold is greater than or equal to the threshold, then the field is considered valid. Related, the fields Merge, and then merge the fields. As a field used in the first stage of querying;

[0029] like If the value is less than the relevance threshold, then the field is considered... If they are irrelevant, then the field will be... As a field for the second stage of query;

[0030] Use the shard key field in the current query conditions as the first-stage query field;

[0031] When querying the database using the current query conditions, the first stage query fields are used to query the database first. If the query result is empty, the second stage query fields are used to query the database.

[0032] Optionally, the step of performing a multi-objective optimization prediction strategy on the non-sharding key aggregation field based on the historical query dataset to predict the optimal query path for the non-sharding key aggregation field includes:

[0033] The multiple objectives include query execution time C1(p(q')), network transmission cost C2(p(q')), storage access cost C3(p(q')), and computing resource consumption C4(p(q')), where p(q') represents the query path of the current query condition;

[0034] The multi-objective optimization function is obtained as follows:

[0035]

[0036] Use a neural network model to compute a multi-objective optimization function;

[0037] The historical query dataset is used to train the neural network model, and each query path corresponds to the query execution time, network transmission cost, storage access cost, and computing resource consumption.

[0038] Calculate the loss function:

[0039] Where τ(θ) is the loss function, θ refers to the weights and biases of the neural network model, and C1(p(q')), C2(p(q')), C3(p(q')) and C4(p(q')) are the actual query execution time, network transmission cost, storage access cost and computing resource consumption, respectively.

[0040] and These represent the query execution time, network transmission cost, storage access cost, and computing resource consumption output by the neural network model, respectively.

[0041] and These represent the squares of the errors between the actual and predicted values ​​for query execution time, network transmission cost, storage access cost, and computing resource consumption, respectively.

[0042] Optionally, the step of performing a multi-objective optimization prediction strategy on the non-sharding key aggregation field based on the historical query dataset to predict the optimal query path for the non-sharding key aggregation field further includes:

[0043] Optimize the loss function;

[0044] Calculate the partial derivative of the loss function with respect to the parameter θ. Where τ is the loss function;

[0045] Calculate the updated parameters Where, θ (t+1) ,θ (t) ... The gradient of the loss function with respect to the parameter θ. It is the exponentially weighted mean of the gradient, where ε is a constant used to prevent division by zero. For standardized gradients;

[0046] Set an error threshold;

[0047] Calculate the difference between two consecutive loss functions and compare the difference with the error threshold;

[0048] If the difference is less than the error threshold, stop optimizing the loss function, obtain the optimized parameters, and obtain the optimal query path based on the optimized parameters.

[0049] Optionally, for the historical query dataset, a preprocessing strategy is executed to cache high-frequency non-sharding key fields and aggregate the non-sharding key fields to obtain non-sharding key aggregate fields, including:

[0050] For each query condition and corresponding query path in the historical query dataset, the query result in the cache is recorded as a marked query.

[0051] Get the total number of marked queries in the historical query dataset, and record it as the cache hit count;

[0052] Calculate cache hit rate = number of cache hits / n;

[0053] Get the read cache time;

[0054] Calculate cache lookup time = cache read time * cache hit rate;

[0055] Set a query time threshold;

[0056] The cached query time is compared with the query time threshold. If the cached query time is less than or equal to the query time threshold, the non-sharding key field in the query condition is identified as a high-frequency non-sharding key field.

[0057] Optionally, the step of performing a preprocessing strategy on the historical query dataset, caching high-frequency non-sharding key fields, and aggregating the non-sharding key fields to obtain aggregated non-sharding key fields, further includes:

[0058] Retrieve all high-frequency non-sharding key fields and cache them.

[0059] Retrieve any two non-sharding key fields f from the historical query dataset e1 ,f e2 ;

[0060] Calculate f e1 ,f e2 nonlinear relationship Wherein, P(x e1 ,x e2 ) is a non-sharding key field f e1 ,f e2 The value is x e1 ,x e2 The joint probability distribution, P(x) e1 ),P(x e2 The distribution represents the non-sharding key field f. e1 ,f e2 The marginal probability distribution, where, Calculate the logarithm of the ratio between the joint probability distribution and the marginal probability distributions, used to quantize the field f. e1 ,f e2 The nonlinear relationship between them The joint probability distribution is used to weight the contribution of each logarithmic term, and the summation means that for all x... e1 ,x e2 For each pair of values, calculate the corresponding term. Used to measure two fields f e1 ,f e2 Nonlinear relationship between them;

[0061] Compare I(f) e1 ,f e2 The relationship between I(f) and 0, if I(f) e1 ,f e2 If )≥0, then the non-sharding key field f e1 ,f e2 Perform aggregation to obtain the non-sharding key aggregated field, and store it in the database;

[0062] If I(f) e1 ,f e2 If ) < 0, then the non-sharding key field f is considered to be... e1 ,f e2 They are independent of each other;

[0063] Retrieve the query criteria for this query;

[0064] First, use a cache query to retrieve the high-frequency non-sharding key field from the cache, and compare it with the fields in the query conditions to determine whether the high-frequency non-sharding key field exists in the query conditions.

[0065] If it exists, return the cached result;

[0066] If it does not exist, a database query will be performed, specifically:

[0067] Retrieve the non-sharded key aggregate field from the database and compare it with the fields in the query conditions to determine whether the fields in the query conditions contain the non-sharded key aggregate field.

[0068] If the aggregation field is not a shard key, the result corresponding to the aggregation field is returned.

[0069] If there are no non-sharding key aggregate fields, then retrieve the fields in the query conditions sequentially to query the database.

[0070] Optionally, the execution of the query load calculation strategy to obtain the query load includes:

[0071] Get the complexity U(q) of each query. z ),1≦z≦w, where w is the number of queries currently being made;

[0072] Calculate query load Where, q z This indicates the current z-th query. The complexity of the current w queries is summed up to form the query load.

[0073] A data query system for health records of a large-scale population includes:

[0074] Electronic health record data management module: manages electronic health record data and partitions the data according to the partition key field;

[0075] Hash classification strategy module: Distributes data to different databases according to the sharding key field using a hash algorithm;

[0076] Query processing and optimization module: Simplifies queries through field correlation analysis, query path prediction, and multi-objective optimization;

[0077] Nonlinear Relationship Analysis and Field Aggregation Module: Analyzes the relationships between high-frequency non-sharding key fields, identifies related fields through joint probability distribution, and performs aggregation to optimize queries;

[0078] High-frequency non-sharding key field caching and aggregation module: caches frequently queried fields and performs aggregation;

[0079] Neural Network Training and Multi-Objective Optimization Module: Trains a neural network model using historical query data, optimizes the query path, and balances multiple objectives;

[0080] Query load monitoring and alarm module: calculates query load in real time, monitors system performance, and issues alarms when the load exceeds a set threshold.

[0081] The present invention has the following beneficial effects:

[0082] 1. This data query method for health records of a large population in a specific area effectively assesses system pressure and responds quickly through real-time monitoring and anomaly detection of query load. When the query load exceeds a preset threshold, the system promptly identifies and triggers anomaly warnings. This provides maintenance personnel with early alerts, preventing performance degradation or system crashes under excessive load. By setting load thresholds, the system can dynamically adjust its strategies based on actual load conditions, ensuring both rapid data retrieval and preventing service interruptions due to excessive query pressure. This load monitoring mechanism allows the system to flexibly handle high-concurrency queries and take measures such as alarms, resource expansion, and rate limiting in abnormal situations, thereby ensuring query performance and system stability.

[0083] 2. The data query method for this massive regional population health records employs a hash-based classification strategy to shard electronic health record records. This allows the system to efficiently distribute data across multiple databases, optimizing query response time and resource utilization. The hash-based classification strategy calculates the hash value of the shard key field, ensuring data is evenly distributed across different databases. This strategy effectively avoids excessive load on some databases due to uneven data distribution, thereby improving system parallelism and reducing resource waste during queries. By evenly distributing records from each shard across different databases, the system achieves efficient data storage and access, ensuring the load on each database remains within a certain range, thus avoiding performance bottlenecks. Through the hash algorithm, the system can distribute data across different storage nodes based on the shard key field, alleviating the pressure on some nodes. For massive electronic health record data, hash-based classification not only improves query efficiency but also optimizes storage resources, exhibiting high scalability and adapting to future data growth.

[0084] 3. The data query method for health records of this large-scale population can further optimize the query path and reduce computational costs by performing field correlation analysis. By analyzing the correlation between query conditions, the system can identify which fields are highly correlated and then merge related fields, thereby simplifying the query operation. For example, if multiple fields in the query are highly correlated, the system can merge the query operations, avoiding multiple data scans and improving query efficiency. By reducing redundant operations in the query, the order and method of data access are optimized, avoiding unnecessary computational overhead.

[0085] 4. The data query method for health records of a large-scale population utilizes a multi-objective optimization prediction strategy. This strategy achieves a balance among multiple optimization objectives, ultimately predicting the optimal query path. The strategy considers multiple objectives, including query execution time, network transmission cost, storage access cost, and computational resource consumption, ensuring optimal balance across these dimensions. Using a neural network model for multi-objective optimization allows the system to learn from historical query datasets, making query path prediction more intelligent and accurate. By training the neural network model, the system can combine the prediction model to output the optimal query path during actual queries, minimizing resource waste. Through multi-objective optimization, the performance of the query system is comprehensively improved, providing the best query solutions under varying query demands, thus ensuring efficient and stable system operation.

[0086] 5. In the data query method for health records of this large-scale population, optimizing the loss function is a core step in training the neural network model during multi-objective optimization. By calculating the loss function and performing backpropagation, the neural network can adjust the weights to make the predicted values ​​closer to the actual values, thereby improving the accuracy of query path prediction. In multi-objective optimization, the loss function considers not only query execution time but also multiple dimensions such as network transmission cost, storage access cost, and computational resource consumption. During optimization, the system gradually adjusts the network model by calculating the partial derivatives of the loss function with respect to the parameters until the error threshold is met. This allows the system to continuously learn and optimize to improve the accuracy of query path prediction. The optimization process of the loss function ensures a balance between various objectives, avoiding unbalanced utilization of system resources due to over-optimization of one objective.

[0087] 6. For the data query method of this ultra-large area population health records, this solution effectively improves query response speed by employing caching and aggregation strategies for frequently queried non-sharding key fields. By storing query results for frequently queried fields in the cache, the system avoids duplicate queries to the database, reducing database query pressure. Furthermore, the system periodically aggregates frequently queried fields, pre-calculating and storing results to reduce computational load during real-time queries. The caching mechanism further reduces query time and optimizes query performance by improving cache hit rate. Through caching and aggregation mechanisms, the frequency of database access is effectively reduced, alleviating database pressure and avoiding performance bottlenecks caused by high-concurrency queries.

[0088] 7. The data query method for health records of this large-scale population utilizes the calculation of non-linear relationships between high-frequency non-sharding key fields to further optimize data aggregation strategies. By identifying which high-frequency fields have significant correlations, the system can merge related fields during queries, reducing the number of queries and data processing time. Through the calculation of joint probability distributions, the system can automatically assess the non-linear relationships between fields, thereby determining which fields can be merged and which need to be queried separately. The system automatically adjusts the query strategy based on the relevance of actual data, avoiding redundant query operations. Through in-depth analysis of the non-linear relationships between fields, the system can make more intelligent and accurate decisions, thereby improving overall query performance and resource utilization efficiency.

[0089] 8. The data query method for health records of this large-scale population allows the system to assess the current query pressure in real time by calculating the query load and dynamically adjust strategies based on query complexity. The query load calculation considers the complexity of each query, helping the system determine whether load balancing, rate limiting, or other measures are necessary. By obtaining the complexity of each query, the system can accurately assess the load and respond appropriately. It provides a tool for quantifying query load, enabling the system to monitor query pressure in real time and dynamically adjust based on query complexity. Through precise query load calculation, the system can avoid overload or performance bottlenecks, improving system stability and response speed. Attached Figure Description

[0090] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0091] Figure 2 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0092] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0093] Example 1, refer to Figure 1 A method for querying health records of a large-scale population, including:

[0094] S1. Define the electronic health record dataset D = {d1, d2, ..., d...} n}, where d iThis represents the i-th electronic health record. Each electronic health record contains a fragment key field and a non-fragment key field. n is the number of electronic health record records, and i is a positive integer.

[0095] In this embodiment, the sharding key field determines the distribution of electronic health record records and directly participates in the sharding rules. It is typically a high-frequency query field or a primary key field. Non-sharding key fields are other fields besides the sharding key field, mainly used to store business data and do not affect the sharding distribution of data.

[0096] S2. Based on the sharding key field and the electronic health record dataset, execute a hash classification strategy to classify the electronic health record records into different databases;

[0097] The database is used to store electronic health record datasets;

[0098] S3. Obtain the historical query dataset H = {(q1,p1),(q2,p2),…,(q v ,p v )}, where each historical query data includes the query path p t and query condition q t =(f1,f2,…,f k ), f1, f2…, f k These are the fields used in the query conditions, f1, f2, ..., f k Includes sharding key fields and non-sharding key fields, k is the number of fields, and v is the number of historical query data;

[0099] Obtain the query conditions, perform field relevance analysis, generate a field relevance matrix, and simplify the query;

[0100] S4. For the historical query dataset, execute a preprocessing strategy to cache the high-frequency non-sharding key fields and aggregate the non-sharding key fields to obtain the non-sharding key aggregate fields.

[0101] S5. Based on the historical query dataset, perform a multi-objective optimization prediction strategy on the non-sharding key aggregation field to predict the optimal query path for the non-sharding key aggregation field.

[0102] S6. Execute the query load calculation strategy to obtain the query load;

[0103] Set a load threshold;

[0104] The query load is compared with the load threshold. If the query load is greater than or equal to the load threshold, an exception is reported.

[0105] By monitoring query load in real time and detecting anomalies, this solution can effectively assess system pressure and react quickly. When the query load exceeds a preset threshold, the system will promptly identify and trigger anomaly warnings. This provides maintenance personnel with early alerts, preventing performance degradation or system crashes under excessive load. By setting load thresholds, the system can dynamically adjust its strategies based on actual load conditions, ensuring fast data retrieval while preventing service interruptions due to excessive query pressure. Using this load monitoring mechanism, the system can flexibly handle high-concurrency queries and take measures such as alerts, resource expansion, and rate limiting in abnormal situations, thereby ensuring query performance and system stability.

[0106] The step involves performing a hash classification strategy based on the sharding key field and the electronic health record dataset to classify electronic health record records into different databases, including:

[0107] The sharding key field set S = {s1, s2, ..., s...} is composed of the sharding key field of each electronic health record in the electronic health record dataset. m}, where s o For electronic health record recording d o The sharding key field, where 1 ≤ o ≤ m, and m is a positive integer;

[0108] Calculate the hash value h(s) of the shard key field. o );

[0109] Get the number of databases, 'a', where 'a' is a positive integer;

[0110] Calculate B j =h(s o )moda, where B j This represents the j-th database, used to store h(s) o The electronic health record record of moda = j, where mod is the modulo operator.

[0111] By employing a hash-based classification strategy to shard electronic health record records, the system can efficiently distribute data across multiple databases, optimizing query response time and resource utilization. The hash-based classification strategy ensures data is evenly distributed across different databases by calculating the hash value of the shard key field. This strategy effectively avoids excessive load on some databases due to uneven data distribution, thereby improving system parallelism and reducing resource waste during queries. By evenly distributing records from each shard across different databases, the system achieves efficient data storage and access, ensuring that the load on each database remains within a certain range, avoiding performance bottlenecks. Through the hash algorithm, the system can distribute data across different storage nodes based on the shard key field, alleviating the pressure on some nodes. For massive amounts of electronic health record data, hash-based classification not only improves query efficiency but also optimizes storage resources, exhibiting high scalability and adapting to future data growth.

[0112] The process of obtaining the current query conditions, performing field relevance analysis, generating a field relevance matrix, and simplifying the query includes:

[0113] Get the query conditions q' = (f1, f2, ..., f k );

[0114] Let the correlation matrix R be defined, and the elements in the correlation matrix R be defined. Representation field and fields A measure of the correlation between them;

[0115] in, fields respectively The value in the l-th electronic health record record, fields respectively The mean;

[0116]

[0117] Set a correlation threshold;

[0118] like If the relevance threshold is greater than or equal to the threshold, then the field is considered valid. Related, the fields Merge, and then merge the fields. As a field used in the first stage of querying;

[0119] like If the value is less than the relevance threshold, then the field is considered... If they are irrelevant, then the field will be... As a field for the second stage of query;

[0120] Use the shard key field in the current query conditions as the first-stage query field;

[0121] When querying the database using the current query conditions, the first stage query fields are used to query the database first. If the query result is empty, the second stage query fields are used to query the database.

[0122] Performing field relevance analysis can further optimize query paths and reduce computational costs. By analyzing the correlation between query conditions, the system can identify which fields are highly correlated and then merge related fields, thereby simplifying query operations. For example, if multiple fields in a query are highly correlated, the system can merge query operations, avoiding multiple data scans and improving query efficiency. The generation of the relevance matrix is ​​automated, requiring no manual intervention, and can be dynamically adjusted for each query to ensure the optimal combination of query conditions. By reducing redundant operations in queries, the order and method of data access are optimized, avoiding unnecessary computational overhead.

[0123] The step of performing a multi-objective optimization prediction strategy on the non-sharding key aggregation field based on the historical query dataset to predict the optimal query path for the non-sharding key aggregation field includes:

[0124] The multiple objectives include query execution time C1(p(q')), network transmission cost C2(p(q')), storage access cost C3(p(q')), and computing resource consumption C4(p(q')), where p(q') represents the query path of the current query condition;

[0125] The multi-objective optimization function is obtained as follows:

[0126]

[0127] Use a neural network model to compute a multi-objective optimization function;

[0128] The historical query dataset is used to train the neural network model, and each query path corresponds to the query execution time, network transmission cost, storage access cost, and computing resource consumption.

[0129] Calculate the loss function:

[0130]

[0131] Where τ(θ) is the loss function, θ refers to the weights and biases of the neural network model, and C1(p(q')), C2(p(q')), C3(p(q')) and C4(p(q')) are the actual query execution time, network transmission cost, storage access cost and computing resource consumption, respectively.

[0132] and These represent the query execution time, network transmission cost, storage access cost, and computing resource consumption output by the neural network model, respectively.

[0133] and These represent the squares of the errors between the actual and predicted values ​​for query execution time, network transmission cost, storage access cost, and computing resource consumption, respectively.

[0134] In this embodiment, the existing neural network model can be any one of feedforward neural networks, convolutional neural networks, recurrent neural networks, or long short-term memory networks.

[0135] By implementing a multi-objective optimization prediction strategy, the system achieves a balance among multiple optimization objectives, ultimately predicting the optimal query path. This strategy considers multiple objectives, including query execution time, network transmission cost, storage access cost, and computational resource consumption, ensuring that the query achieves an optimal balance across these dimensions. Using neural networks for multi-objective optimization allows the system to learn from historical query datasets, making query path prediction more intelligent and accurate. By training the neural network model, the system can combine the prediction model to output the optimal query path during actual queries, minimizing resource waste. Through multi-objective optimization, the performance of the query system can be comprehensively improved, providing the best query solutions under varying query demands, thereby ensuring the system's efficient and stable operation.

[0136] The step of performing a multi-objective optimization prediction strategy on the non-sharding key aggregation field based on the historical query dataset to predict the optimal query path for the non-sharding key aggregation field also includes:

[0137] Optimize the loss function;

[0138] Calculate the partial derivative of the loss function with respect to the parameter θ. Where τ is the loss function;

[0139] Calculate the updated parameters Where, θ (t+1) ,θ (t) ... The gradient of the loss function with respect to the parameter θ. It is the exponentially weighted mean of the gradient, where ε is a constant used to prevent division by zero. For standardized gradients;

[0140] Set an error threshold;

[0141] Calculate the difference between two consecutive loss functions and compare the difference with the error threshold;

[0142] If the difference is less than the error threshold, stop optimizing the loss function, obtain the optimized parameters, and obtain the optimal query path based on the optimized parameters.

[0143] In multi-objective optimization, optimizing the loss function is a core step in training a neural network model. By calculating the loss function and performing backpropagation, the neural network can adjust its weights to make the predicted values ​​closer to the actual values, thereby improving the accuracy of query path prediction. In multi-objective optimization, the loss function considers not only query execution time but also multiple dimensions such as network transmission cost, storage access cost, and computational resource consumption. During optimization, the system calculates the partial derivatives of the loss function with respect to the parameters and gradually adjusts the network model until an error threshold is met. This allows the system to continuously learn and optimize, improving the accuracy of query path prediction. The optimization process of the loss function ensures a balance between the various objectives, avoiding unbalanced utilization of system resources due to over-optimization of any one objective.

[0144] The process involves performing a preprocessing strategy on the historical query dataset, caching high-frequency non-sharding key fields, and aggregating these fields to obtain aggregated non-sharding key fields, including:

[0145] For each query condition and corresponding query path in the historical query dataset, the query result in the cache is recorded as a marked query.

[0146] Get the total number of marked queries in the historical query dataset, and record it as the cache hit count;

[0147] Calculate cache hit rate = number of cache hits / n;

[0148] Get the read cache time;

[0149] Calculate cache lookup time = cache read time * cache hit rate;

[0150] Set a query time threshold;

[0151] The cached query time is compared with the query time threshold. If the cached query time is less than or equal to the query time threshold, the non-sharding key field in the query condition is identified as a high-frequency non-sharding key field.

[0152] For frequently queried non-sharded key fields, this solution significantly improves query response speed through caching and aggregation strategies. By storing query results for frequently queried fields in the cache, the system avoids duplicate database queries, reducing database query pressure. Furthermore, the system periodically aggregates frequently queried fields, pre-calculating and storing results to reduce computational load during real-time queries. The caching mechanism further reduces query time and optimizes query performance by improving cache hit rate. Through caching and aggregation mechanisms, the frequency of database access is significantly reduced, alleviating database pressure and avoiding performance bottlenecks caused by high-concurrency queries.

[0153] The process of performing a preprocessing strategy on the historical query dataset, caching high-frequency non-sharding key fields, and aggregating the non-sharding key fields to obtain aggregated non-sharding key fields, also includes:

[0154] Retrieve all high-frequency non-sharding key fields and cache them.

[0155] Retrieve any two non-sharding key fields f from the historical query dataset e1 ,f e2 ;

[0156] Calculate f e1 ,f e2 nonlinear relationship Wherein, P(x e1 ,x e2 ) is a non-sharding key field f e1 ,f e2 The value is x e1 ,x e2 The joint probability distribution, P(x) e1 ),P(x e2 The distribution represents the non-sharding key field f. e1 ,f e2 The marginal probability distribution, where, Calculate the logarithm of the ratio between the joint probability distribution and the marginal probability distributions, used to quantize the field f. e1 ,f e2 The nonlinear relationship between them The joint probability distribution is used to weight the contribution of each logarithmic term, and the summation means that for all x... e1 ,x e2 For each pair of values, calculate the corresponding term. Used to measure two fields f e1 ,f e2 Nonlinear relationship between them;

[0157] Compare I(f) e1 ,f e2The relationship between I(f) and 0, if I(f) e1 ,f e2 If )≥0, then the non-sharding key field f e1 ,f e2 Perform aggregation to obtain the non-sharding key aggregated field, and store it in the database;

[0158] If I(f) e1 ,f e2 If ) < 0, then the non-sharding key field f is considered to be... e1 ,f e2 They are independent of each other;

[0159] Retrieve the query criteria for this query;

[0160] First, use a cache query to retrieve the high-frequency non-sharding key field from the cache, and compare it with the fields in the query conditions to determine whether the high-frequency non-sharding key field exists in the query conditions.

[0161] If it exists, return the cached result;

[0162] If it does not exist, a database query will be performed, specifically:

[0163] Retrieve the non-sharded key aggregate field from the database and compare it with the fields in the query conditions to determine whether the fields in the query conditions contain the non-sharded key aggregate field.

[0164] If the aggregation field is not a shard key, the result corresponding to the aggregation field is returned.

[0165] If there are no non-sharding key aggregate fields, then retrieve the fields in the query conditions sequentially to query the database.

[0166] By calculating the non-linear relationships between high-frequency non-sharding key fields, the system can further optimize data aggregation strategies. By identifying which high-frequency fields have significant correlations, the system can merge related fields during queries, reducing the number of queries and data processing time. Through the calculation of joint probability distributions, the system can automatically evaluate the non-linear relationships between fields, thereby determining which fields can be merged and which fields need to be queried separately. By deeply analyzing the non-linear relationships between fields, the system can make more intelligent and accurate decisions, thereby improving the overall query performance and resource utilization efficiency.

[0167] The query load calculation strategy is executed to obtain the query load, including:

[0168] Get the complexity U(q) of each query. z ),1≦z≦w, where w is the number of queries currently being made;

[0169] Calculate query load Where, qz This indicates the current z-th query. The complexity of the current w queries is summed up to form the query load.

[0170] By calculating query load, the system can assess the current query pressure in real time and dynamically adjust its strategies based on query complexity. The query load calculation takes into account the complexity of each query, helping the system determine whether load balancing, rate limiting, or other measures are necessary. By obtaining the complexity of each query, the system can accurately assess the load and react appropriately. It provides a tool for quantifying query load, enabling the system to monitor query pressure in real time and dynamically adjust based on query complexity. Through precise query load calculation, the system can avoid overload or performance bottlenecks, improving system stability and response speed.

[0171] Example 2, refer to Figure 2 A data query system for health records of a large-scale population, comprising:

[0172] Electronic health record data management module: manages electronic health record data and partitions the data according to the partition key field;

[0173] Hash classification strategy module: Distributes data to different databases according to the sharding key field using a hash algorithm;

[0174] Query processing and optimization module: Simplifies queries through field correlation analysis, query path prediction, and multi-objective optimization;

[0175] Nonlinear Relationship Analysis and Field Aggregation Module: Analyzes the relationships between high-frequency non-sharding key fields, identifies related fields through joint probability distribution, and performs aggregation to optimize queries;

[0176] High-frequency non-sharding key field caching and aggregation module: caches frequently queried fields and performs aggregation;

[0177] Neural Network Training and Multi-Objective Optimization Module: Trains a neural network model using historical query data, optimizes the query path, and balances multiple objectives;

[0178] Query load monitoring and alarm module: calculates query load in real time, monitors system performance, and issues alarms when the load exceeds a set threshold.

[0179] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0180] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for querying health records of a population in a very large area, characterized in that: include: S1. Define the electronic health record dataset. ,in, This represents the i-th electronic health record. Each electronic health record contains a fragment key field and a non-fragment key field. n is the number of electronic health record records, and i is a positive integer. S2. Based on the sharding key field and the electronic health record dataset, execute a hash classification strategy to classify the electronic health record records into different databases; The database is used to store electronic health record datasets; S3. Obtain historical query dataset Each historical query record includes the query path. and query conditions , These are fields used for query conditions. This includes sharding key fields and non-sharding key fields, where k is the number of fields and v is the number of historical query data. v is a positive integer; Obtain the query conditions, perform field relevance analysis, generate a field relevance matrix, and simplify the query; Get the query conditions ; Let the correlation matrix R be defined, and the elements in the correlation matrix R be defined. Representation field and fields A measure of the correlation between them; ,in, fields respectively In the Values ​​recorded in an electronic health record. fields respectively The mean; ； ； Set a correlation threshold; like If the relevance threshold is greater than or equal to the threshold, then the field is considered valid. Related, the fields Merge, and then merge the fields. As a field used in the first stage of querying; like If the value is less than the relevance threshold, then the field is considered... If they are irrelevant, then the field will be... As a field for the second stage of query; Use the shard key field in the current query conditions as the first-stage query field; When querying the database using the current query conditions, the first stage query fields are used to query the database first. If the query result is empty, the second stage query fields are used to query the database. S4. For the historical query dataset, execute a preprocessing strategy to cache the high-frequency non-sharding key fields and aggregate the non-sharding key fields to obtain the non-sharding key aggregate fields. Retrieve the query criteria for this query; First, use a cache query to retrieve the high-frequency non-sharding key field from the cache, and compare it with the fields in the query conditions to determine whether the high-frequency non-sharding key field exists in the query conditions. If it exists, return the cached result; If it does not exist, a database query will be performed, specifically: Retrieve the non-sharded key aggregate field from the database and compare it with the fields in the query conditions to determine whether the fields in the query conditions contain the non-sharded key aggregate field. If the aggregation field is not a shard key, the result corresponding to the aggregation field is returned. If there is no non-sharding key aggregation field, then retrieve the fields in the query conditions sequentially to query the database; S5. Based on the historical query dataset, perform a multi-objective optimization prediction strategy on the non-sharding key aggregation field to predict the optimal query path for the non-sharding key aggregation field. The multiple objectives include query execution time. Network transmission costs Storage access costs and computational resource consumption ,in, Indicates the query path for the current query condition; S6. Execute the query load calculation strategy to obtain the query load; Set a load threshold; The query load is compared with the load threshold. If the query load is greater than or equal to the load threshold, an exception is reported.

2. The data query method for health records of a large-scale population as described in claim 1, characterized in that: The step involves performing a hash classification strategy based on the sharding key field and the electronic health record dataset to classify electronic health record records into different databases, including: The sharding key field set is composed of the sharding key field of each electronic health record in the electronic health record dataset. ,in, For electronic health record records The sharding key field, where, m is a positive integer; Calculate the hash value of the shard key field ; Get the number of databases , It is a positive integer; calculate ,in, This represents the j-th database, used for storing... The electronic health record records, where the mod operator is the modulo operator.

3. The data query method for health records of a large-scale population as described in claim 2, characterized in that: The step of performing a multi-objective optimization prediction strategy on the non-sharding key aggregation field based on the historical query dataset to predict the optimal query path for the non-sharding key aggregation field includes: The multi-objective optimization function is obtained as follows: ； Use a neural network model to compute a multi-objective optimization function; The historical query dataset is used to train the neural network model, and each query path corresponds to the query execution time, network transmission cost, storage access cost, and computing resource consumption. Calculate the loss function: ; in, For loss function, This refers to the weights and biases of a neural network model. , , and These are the actual query execution time, network transmission cost, storage access cost, and computing resource consumption, respectively. , , and These represent the query execution time, network transmission cost, storage access cost, and computing resource consumption output by the neural network model, respectively. , , and These represent the squares of the errors between the actual and predicted values ​​for query execution time, network transmission cost, storage access cost, and computing resource consumption, respectively.

4. The data query method for health records of a large-scale population as described in claim 3, characterized in that: The step of performing a multi-objective optimization prediction strategy on the non-sharding key aggregation field based on the historical query dataset to predict the optimal query path for the non-sharding key aggregation field also includes: Optimize the loss function; Calculate the loss function with respect to parameters partial derivatives ,in, It is a loss function; Calculate the updated parameters ,in, Let be the neural network parameters for the t-th and t+1-th iterations, respectively. It's the learning rate, which controls the step size for each update. For the loss function with respect to parameters gradient, It is the exponentially weighted mean of the gradient. It is a constant used to prevent division by zero. For standardized gradients; Set an error threshold; Calculate the difference between two consecutive loss functions and compare the difference with the error threshold; If the difference is less than the error threshold, stop optimizing the loss function, obtain the optimized parameters, and obtain the optimal query path based on the optimized parameters.

5. The data query method for health records of a large-area population according to claim 1, characterized in that: The process involves performing a preprocessing strategy on the historical query dataset, caching high-frequency non-sharding key fields, and aggregating these fields to obtain aggregated non-sharding key fields, including: For each query condition and corresponding query path in the historical query dataset, the query results in the cache are recorded as marked queries; Get the total number of marked queries in the historical query dataset, and record it as the cache hit count; Calculate cache hit rate = number of cache hits / n; Get the read cache time; Calculate cache lookup time = cache read time * cache hit rate; Set a query time threshold; The cached query time is compared with the query time threshold. If the cached query time is less than or equal to the query time threshold, the non-sharding key field in the query condition is identified as a high-frequency non-sharding key field.

6. The data query method for health records of a large-area population according to claim 5, characterized in that: The process of performing a preprocessing strategy on the historical query dataset, caching high-frequency non-sharding key fields, and aggregating the non-sharding key fields to obtain aggregated non-sharding key fields, also includes: Retrieve all high-frequency non-sharding key fields and cache them. Retrieve any two non-sharding key fields from the historical query dataset ; calculate nonlinear relationship ,in, Is it a non-sharding key field? Values The joint probability distribution, Distribution represents non-sharding key fields The marginal probability distribution, where, Calculate the logarithm of the ratio between the joint probability distribution and the marginal probability distributions to quantize the field. The nonlinear relationship between them The joint probability distribution is used to weight the contribution of each logarithmic term, and the summation means that for all... For each pair of values, calculate the corresponding term. Used to measure two fields Nonlinear relationship between them; Compare The relationship between the magnitude of 0 and 0, if Then the non-sharding key field Perform aggregation to obtain the non-sharding key aggregated field, and store it in the database; like Then it is determined that the non-sharding key field They are independent of each other.

7. The data query method for health records of a large-area population according to claim 1, characterized in that: The query load calculation strategy is executed to obtain the query load, including: Get the complexity U(q) of each query. z ),1≦z≦w, where w is the number of queries currently being made; Calculate query load , where q z This indicates the current z-th query. The complexity of the current w queries is summed up to form the query load.

8. A data query system for health records of a large-scale population, applied to the data query method for health records of a large-scale population as described in any one of claims 1-7, characterized in that, include: Electronic health record data management module: manages electronic health record data and partitions the data according to the sharding key field; Hash classification strategy module: Distributes data to different databases according to the sharding key field using a hash algorithm; Query processing and optimization module: Simplifies queries through field correlation analysis, query path prediction, and multi-objective optimization; Nonlinear Relationship Analysis and Field Aggregation Module: Analyzes the relationships between non-sharding key fields and performs aggregation to optimize queries; High-Frequency Non-Sharding Key Field Caching and Aggregation Module: Caches and aggregates frequently queried fields; Neural Network Training and Multi-Objective Optimization Module: Trains a neural network model using historical query data, optimizes the query path, and balances multiple objectives; Query load monitoring and alarm module: calculates query load in real time, monitors system performance, and issues alarms when the load exceeds a set threshold.

Images