Data query method, device and electronic equipment

Abstract

The application provides a data query method and device and electronic equipment, and relates to the technical field of infrastructure and IT support, wherein the method comprises the following steps: receiving a first request of a user; in response to the first request, determining at least two storage nodes corresponding to the first request, wherein the at least two storage nodes comprise heterogeneous storage nodes distributed in different regions; respectively determining first weights of the at least two storage nodes, wherein the first weights are determined according to performance indexes of the at least two storage nodes; and distributing the first request to a first storage node according to the first weights, wherein the at least two storage nodes comprise the first storage node. According to the embodiment of the application, the weights of the plurality of storage nodes are determined according to the performance indexes of the storage nodes, and the user query request is distributed according to the weights, so that the query service can be provided according to the real-time running performance of the storage nodes, and the user query efficiency can be improved.

Description

Technical Field

[0001] This application relates to the fields of infrastructure and IT support technology, and in particular to a data query method, apparatus and electronic device. Background Technology

[0002] With the rapid development of information technology and the dramatic increase in data volume, distributed storage technology has emerged. Distributed storage achieves distributed processing of file storage and access by distributing file data across multiple server nodes, effectively solving many limitations of traditional centralized file systems.

[0003] However, in current distributed storage systems, data is usually stored in multiple storage systems provided by different vendors. Due to the differences in metadata formats and semantics used by different vendors, when users query data, they need to use the statically configured Domain Name System (DNS) routing method, which results in low data query efficiency. Summary of the Invention

[0004] This application provides a data query method, apparatus, and electronic device to solve the problem of low data query efficiency when querying through statically configured Domain Name System routing.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a data query method, the method comprising:

[0007] Receive the user's first request, which is used to query data on the storage node;

[0008] In response to the first request, at least two storage nodes corresponding to the first request are identified, the at least two storage nodes including heterogeneous storage nodes distributed in different regions;

[0009] A first weight is determined for each of the at least two storage nodes, and the first weight is determined based on the performance indicators of the at least two storage nodes;

[0010] According to the first weight, the first request is distributed to the first storage node, wherein the at least two storage nodes include the first storage node.

[0011] Secondly, embodiments of this application provide a data query device, the device comprising:

[0012] The receiving module is used to receive the user's first request, which is used to query the data of the storage node.

[0013] A first determining module is configured to, in response to the first request, determine at least two storage nodes corresponding to the first request, wherein the at least two storage nodes include heterogeneous storage nodes distributed in different regions;

[0014] The second determining module is used to determine the first weight of each of the at least two storage nodes, wherein the first weight is determined based on the performance indicators of the at least two storage nodes;

[0015] A distribution module is configured to distribute the first request to a first storage node according to the first weight, wherein the at least two storage nodes include the first storage node.

[0016] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the data query method described in the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data query method described in the first aspect.

[0018] Fifthly, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data query method as described in the first aspect.

[0019] In this embodiment, a first request from a user is received, the first request being for querying data on storage nodes. In response to the first request, at least two storage nodes corresponding to the first request are determined, the at least two storage nodes including heterogeneous storage nodes distributed in different regions. A first weight is determined for each of the at least two storage nodes, the first weight being determined based on the performance indicators of the at least two storage nodes. According to the first weight, the first request is distributed to a first storage node, the at least two storage nodes including the first storage node. Through this method, when a query request corresponds to multiple storage nodes, the weights of the multiple storage nodes are determined based on their performance indicators, and the user query request is distributed according to these weights. This allows for real-time provision of query services based on the operating performance of the storage nodes, thereby improving user query efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a data query method provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the architecture of a data query system provided in an embodiment of this application;

[0023] Figure 3 This is a data query diagram provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] This application provides a data query method, apparatus, and electronic device to solve the problem of low data query efficiency when querying through statically configured DNS routing.

[0028] See Figure 1 , Figure 1 This is a flowchart of a data query method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0029] Step 101: Receive the user's first request, which is used to query the data of the storage node;

[0030] Step 102: In response to the first request, determine at least two storage nodes corresponding to the first request, wherein the at least two storage nodes include heterogeneous storage nodes distributed in different regions;

[0031] Step 103: Determine the first weight of each of the at least two storage nodes, wherein the first weight is determined based on the performance indicators of the at least two storage nodes;

[0032] Step 104: Distribute the first request to the first storage node according to the first weight, wherein the at least two storage nodes include the first storage node.

[0033] The first request can be a query input by the user to retrieve data stored on a storage node. Based on the first request, at least two corresponding storage nodes can be obtained, meaning that the data is stored on at least two storage nodes, and the user can query the data they need through any one of these at least two storage nodes.

[0034] In some implementations, the logical identifier in the user request is parsed into the storage service address of a specific vendor, and the corresponding storage node is determined based on the storage service address.

[0035] In some implementations, the user's IP address or region identifier is obtained from the gateway layer, and the storage node is determined based on historical query logs and geographical location matching degree.

[0036] At least two storage nodes include heterogeneous storage nodes from different vendors located in different regions. A heterogeneous storage node can be understood as a storage node that differs from another storage node in at least one of the following: architecture, data format, or data semantics. For example, it includes a storage node from vendor A located in region 1 and a storage node from vendor B located in region 2.

[0037] The first storage node can be one or more storage nodes from vendor A. The second storage node can be one or more storage nodes from vendor B. When the first storage node includes multiple storage nodes, the performance metric of the first storage node can be the average or sum of the performance metrics of the multiple storage nodes.

[0038] In some implementations, a global metadata directory among storage nodes can be pre-built and maintained, including the network address, location, performance information, etc. of the storage nodes. Based on this global metadata directory, performance metric information of at least two storage nodes can be quickly obtained, and a first weight of at least two storage nodes can be determined based on the performance metrics, thereby allocating requests according to the first weight.

[0039] In some implementations, a pre-trained model is used to control the first weight of the storage node;

[0040] In some implementations, a first weight of a storage node is determined based on its data transfer cost and disaster recovery rating, and requests are then allocated according to this first weight.

[0041] In some implementations, the distribution path of requests is dynamically adjusted based on metadata and real-time performance metrics provided by the metadata governance tool (Gravitino).

[0042] In some implementations, after determining the first weights of multiple storage nodes, user query requests can be allocated according to the determined first weights within a preset time period. Furthermore, at each preset time interval, the first weights of the multiple storage nodes are recalculated based on the user's query requests.

[0043] For example, upon receiving a first request, two storage nodes are determined based on the data queried in the first request, and the first weights of the two storage nodes are determined to be 70% and 30% respectively based on the performance information of the two storage nodes. The first request is then allocated to the corresponding storage node according to these weights.

[0044] If a second request is received within a preset time period after the first weight is determined, and the data being queried is also from the two storage nodes mentioned above, the second request can be directly allocated according to the weight mentioned above.

[0045] In this embodiment of the application, when a query request corresponds to multiple storage nodes, a first weight of the multiple storage nodes is determined based on the performance indicators of the storage nodes, and the user query request is allocated according to the first weight. This can provide query services in real time based on the operating performance of the storage nodes, thereby improving the user's query efficiency.

[0046] Optionally, determining at least two storage nodes corresponding to the first request includes:

[0047] Based on the first request, obtain the network address information of at least two storage nodes;

[0048] Based on a pre-built global metadata directory, the at least two storage nodes are determined according to the network address information. The global metadata directory includes a mapping relationship between storage nodes and network address information.

[0049] The network address information can be the network service address information of the storage nodes. The global metadata directory can include metadata for heterogeneous storage nodes, including network address information, storage node information, performance information, etc., for each storage node. The Gravitino metadata federated governance layer, as the global metadata hub, can be used for unified management, governance, and optimization of cross-vendor metadata. The unified DNS intelligent routing layer, as the global traffic entry point, is responsible for intelligent request distribution and load balancing. Specifically, the unified DNS intelligent routing layer includes:

[0050] DNS resolution module: used to resolve user-submitted query requests, extract the logical identifier of the request, such as virtual path or domain name, and determine the corresponding physical storage address (i.e., network address information) based on the pre-established mapping relationship.

[0051] Dynamic routing strategy module: This module uses a Q-Learning model to train the optimal distribution path and dynamically adjusts the DNS weights based on the real-time performance of storage nodes (such as latency and throughput). Specifically, the module collects performance metric data from storage nodes, calculates a comprehensive node score (to reflect node performance) at preset time intervals (such as 5 seconds), and dynamically adjusts the DNS weights based on the node scores.

[0052] Fault self-healing and load balancing module: Supports primary and backup storage node switching and weight allocation to ensure high system availability. When a storage node from one vendor becomes unavailable, it automatically switches to the backup vendor and updates the Gravitino metadata status. This module uses a sliding window (such as the average of the past 3 scores) to avoid drastic weight fluctuations and sets a minimum weight (such as 5%) to prevent nodes from being completely removed.

[0053] As one implementation method, the implementation process of each module is shown in Table 1 below:

[0054] Table 1

[0055] Hostname resolution module User query requests (e.g., SELECT * FROM gravitino: / / db.table1) 1. Extract the logical identifier (virtual path db.table1) 2. Query the preset mapping table 3. Convert to a physical storage address (e.g., hdfs: / / vendorA / path) Physical address + logical identifier Dynamic routing strategy module Physical address + logical identifier 1. Retrieve Gravitino metadata (data distribution, load) 2. Q-Learning model training path 3. Calculate node scores every 5 seconds (Stotal formula) 4. Dynamically adjust weights (Wi formula) Routing rule table (e.g., vendor A has a weight of 70%, vendor B has a weight of 30%). Fault self-healing and load balancing module Routing rule table 1. Health check (HTTP / health) 2. Sliding window weight smoothing (e.g., average of the past 3 times) 3. Primary / standby failover (e.g., weight resets to zero after 3 consecutive failures) 4. Minimum weight protection (e.g., ≥5%) Distribution instructions + metadata update events

[0056] In the above process, the user enters a query request: SELECT * FROM gravitino: / / prod_db.orders WHERE date>20230101

[0057] Logical identifier extraction: virtual path prod_db.orders;

[0058] Generating physical addresses based on virtual paths:

[0059] Mapped to vendor A's address hdfs: / / clusterA / orders;

[0060] Mapped to vendor B's address s3: / / bucketB / orders (backup);

[0061] The dynamic routing strategy module optimizes paths based on physical addresses.

[0062] The physical address undergoes a three-stage evolution at this layer, as shown in Table 2:

[0063] Table 2

[0064] Initial mapping Static preset address The mapping table of the Hostname resolution module After route optimization Weight dynamic adjustment address Wi formula for dynamic routing policy module After fault switching Alternate address activation Fault self-healing module master / slave switching logic

[0065] Optionally, determining the first weights of the at least two storage nodes includes:

[0066] Based on a pre-built global metadata directory, obtain the performance metrics of the at least two storage nodes;

[0067] The performance scores of the at least two storage nodes are determined based on their respective performance metrics, and the performance scores are scores of the operating performance of the storage nodes.

[0068] The first weight of the at least two storage nodes is determined based on their performance scores.

[0069] The metadata global directory can include information, address information, and performance metrics of at least two heterogeneous storage nodes.

[0070] First, obtain the performance metrics of the storage nodes based on the global metadata directory.

[0071] For example, query Gravitino to obtain partition information of the orders table (e.g., region = 'Beijing'), determine the storage node based on the query information, and the corresponding real-time performance indicators, including network latency, throughput, etc.

[0072] In some implementations, a performance score is determined based on performance metrics to obtain a performance score for each storage node, and the performance score can reflect the performance of the storage node.

[0073] After obtaining the performance score, the first weight of each storage node is determined based on the proportion of each storage node's performance score to the total score.

[0074] In some implementations, the first weight of each storage node can be determined based on a pre-defined mapping relationship between performance scores and weights.

[0075] In some implementations, the first weight of each storage node can be determined based on the performance score and historical performance scores.

[0076] By employing the methods described above, distribution instability caused by single performance fluctuations can be prevented, thereby improving distribution stability.

[0077] Optionally, determining the first weight of the at least two storage nodes based on their performance scores includes any of the following:

[0078] Based on the preset correspondence between the performance score and the weight, a first weight corresponding to the performance score of each of the at least two storage nodes is determined;

[0079] A second weight for each storage node is determined based on the proportion of its performance score to the total performance score of the at least two storage nodes. A first weight for each storage node is determined based on its historical weight and the second weight, wherein the historical weight is the historical first weight of each storage node.

[0080] After obtaining the performance score, different weighting methods can be selected depending on the timing:

[0081] The first approach is to initialize the DNS's initial weight. This can be achieved by establishing a storage node weight mapping rule, assigning the initial weight based on the storage node's rating. Specifically, a linear mapping rule is used, mapping the rating range [0, 100] to the weight range [5%, 100%], with higher ratings resulting in greater initial weights. The core of this mapping rule is linear mapping, ensuring the initial weight is proportional to the rating, while setting a lower limit to guarantee system stability.

[0082] Linear mapping mechanism: The score (Si) ranges from 0 to 100 points and is directly and linearly converted to the first weight (Wi), with the first weight fixed in the range of [5%, 100%]. The specific mapping relationship is as follows:

[0083] The lowest score (0 points) corresponds to the minimum weight of 5%.

[0084] The highest score (100 points) corresponds to the highest weight of 100%.

[0085] The rating increases linearly proportionally (e.g., a rating of 50 points has a weight of approximately 52.5%), and the specific rules for this linear increase are adjusted by different users according to their specific needs.

[0086] Based on the above mapping relationship, the first weight corresponding to the performance score of each storage node can be determined.

[0087] The second method is to determine the first weight based on historical weights.

[0088] Taking storage nodes from vendor A and vendor B as examples, a score is calculated for each storage node based on the obtained performance metrics (e.g., vendor A latency = 8ms, vendor B latency = 15ms) (example):

[0089] S_total = 0.4*S_latency + 0.3*S_throughput + 0.3*S_cpu;

[0090] Based on the above-mentioned common knowledge, the final calculation shows that Manufacturer A scored 90 points and Manufacturer B scored 75 points.

[0091] Here, S_latency, S_throughput, and S_cpu represent normalized scores for latency, throughput, and CPU utilization, respectively, sourced from real-time collected performance metrics (via monitoring agents and a central monitoring system). The scoring rules can be predefined by the user. These scores are used to calculate overall performance through a weighted formula, driving dynamic adjustments to DNS weights to achieve efficient and stable load balancing.

[0092] Based on the scores, weights are assigned to obtain the second weights for vendor A and vendor B:

[0093] W_A = 90 / (90+75)*100 ≈ 55%

[0094] W_B = 75 / (90+75)*100 ≈ 45%

[0095] The first weight is calculated based on the second weight and historical weights using a weight smoothing method.

[0096] Historical weights serve as the input source, and a sliding window smooths them to generate intermediate values. The final weights are then allocated based on the real-time status output to optimize the allocation. Specifically, historical weights are calculated every 5 seconds by the dynamic routing strategy module and stored as a time series (e.g., [60%, 50%, 55%]). These values ​​are directly input into the sliding window mechanism. The sliding window takes a specified size (e.g., the past 3 historical weights) and calculates the arithmetic mean (e.g., the sliding window smooths the weights to the average of the past 3 weights). For example, when the window size is 3, the average is (weight 1 + weight 2 + weight 3) / 3.

[0097] Historical weights form the basis of the sliding window calculation. The sliding window reduces noise (such as short-term performance fluctuations) by averaging, generating smoothed weight values ​​(i.e., "sliding window resistance" or average value), providing a stable input for the final allocation to avoid drastic weight fluctuations.

[0098] The final allocation result will be fed back to the system: the distribution instruction is used to request routing (e.g., distribution to vendor A) and triggers metadata update events (e.g., updating Gravitino metadata status). If the weights are adjusted (e.g., switching between primary and backup vendors), the weights calculated in real time will be used as part of the historical weights for the next sliding window calculation.

[0099] Example explanation:

[0100] Historical weights: [60%, 50%, 55%], sliding window mean ≈ 55%.

[0101] Final allocation: Manufacturer A has a weight of 55%, and Manufacturer B has a weight of 45%.

[0102] Assume the weights calculated by the dynamic routing policy module in the last three calculations are [60% (vendor A), 50% (vendor A), 55% (vendor A)]. These are historical weights.

[0103] The sliding window takes the weights from the past 3 times (this is just an example) and calculates the average: (60% + 50% + 55%) / 3 ≈ 55%. This generates a smooth weight of 55% (the median value). By doing so, drastic changes in weights (such as jumping directly from 50% to 60%) can be avoided, preventing distribution instability caused by a single performance fluctuation.

[0104] Optionally, the method further includes at least one of the following:

[0105] If the first storage node fails to perform health checks M times, the first weight of the first storage node is set to 0, and the storage node corresponding to the first request is switched from the first storage node to the second storage node among the at least two storage nodes according to the magnitude of the first weight, where M is an integer greater than 0.

[0106] If the second weight of the first storage node is less than a preset value, the weight value of the second weight of the first storage node is set to the third weight value.

[0107] The M health check failures can be due to performance metrics falling below preset values ​​or health scores falling below preset values. If M consecutive health checks fail, the first weight of the first storage node is set to 0, meaning no requests are allocated.

[0108] For example, based on the aforementioned smoothing weight of 55%, an additional rule is applied: if the health check of vendor A node is successful (vendor A node responds normally, HTTP 200), then 55% is retained; if it fails 3 times in a row, the first weight is reset to zero (switching to vendor B).

[0109] In addition, minimum weight protection is employed. Based on the aforementioned second weight of 55%, an additional rule applies: if the second weight 55% > 5%, no adjustment is needed; however, if the second weight is calculated to be 3%, then the second weight is set to 5%.

[0110] The final weighting is 55% for vendor A and 45% for vendor B (calculated based on a total score of 100%). Additionally, if a switch occurs, a metadata update event is output (e.g., notifying Gravitino that vendor A is unavailable).

[0111] The request is distributed to the appropriate physical storage device, the file is read on the device, and the result is returned.

[0112] If vendor A times out continuously, automatically switch to vendor B and update Gravitino.

[0113] In this way, the average value of the sliding window (smoothing weight) is passed to the final allocation stage as an intermediate value. In the final allocation, the smoothing weight is not directly output, but is adjusted in conjunction with the following real-time rules, such as: the module performs health checks on storage nodes, and if a node fails 3 times in a row, the weight is forcibly set to 0, triggering a switch to a standby node; a weight lower limit (≥5%) is set to prevent nodes from being completely removed due to temporary problems.

[0114] The smoothing weight of the sliding window is the primary basis for the final allocation, but the final allocation will be superimposed with real-time health status and minimum protection rules. For example, the smoothing weight is 55%, but if a node health check fails, the weight may drop to 0; conversely, if the smoothing weight is too low (e.g., below 5%), it will be increased to 5%. This ensures that the final allocation is both smooth and fault-tolerant.

[0115] Optionally, the method further includes:

[0116] Obtain metadata from the at least two storage nodes and construct a global metadata directory based on the metadata, wherein the metadata includes at least one of the data distribution information and performance information of the at least two nodes;

[0117] Listen for metadata events and update the metadata in the global metadata directory based on the metadata events, wherein the metadata events include at least one of metadata change events and addition events.

[0118] The Gravitino metadata federation governance layer serves as the global metadata hub, enabling unified management, governance, and optimization of metadata across vendors. Specifically, it includes:

[0119] Gravitino integrates metadata from multiple storage vendors (such as HDFS and S3) via adapters to build a global metadata catalog. This module uses a distributed database (such as Apache HBase) to store the global metadata catalog, supporting high-concurrency access. Specifically, the module listens for metadata change events in the storage system (such as file and directory creation and deletion, and changes in storage size), and sends messages through the real-time interface module between the systems to trigger Gravitino's metadata update process, ensuring the real-time performance of the global catalog.

[0120] Historical query logs are collected and stored in a data warehouse. Machine learning models and algorithms are used to analyze these logs, pre-caching hotspot metadata and optimizing data lineage tracing. This module can build data lineage graphs, supporting rapid source tracing.

[0121] The system monitors metadata changes (such as new tables or partition adjustments) on each heterogeneous storage node in real time and sends messages through the real-time interface module between systems to trigger automatic updates to the global view. An event-driven synchronization mechanism is used to ensure the real-time performance of the global directory.

[0122] Data is dynamically sharded based on query characteristics, and query tasks are executed in parallel to achieve load balancing. Specifically, the module dynamically allocates shards based on the storage node load to avoid hotspot issues. The coordination mechanism between various functions is as follows:

[0123] (1) Metadata aggregation and dynamic synchronization: event-driven update.

[0124] The global directory output by the aggregation module serves as the base version for the synchronization module. It listens for vendor events (such as the HDFS INotify file creation event), extracts changed content (adding partition date=20230804), and incrementally updates the global directory (modifying only the changed parts to avoid a full refresh).

[0125] (2) Dynamic synchronization to enhanced governance: real-time status empowers intelligent decision-making.

[0126] Bloodline map construction:

[0127] Input: Table change events provided by the synchronization module (such as table A → table B ETL link).

[0128] Output: Cross-vendor lineage map (supports rapid tracing);

[0129] Hot Topic Prediction Linkage:

[0130] The synchronization module transmits high-frequency change table information to the governance module, which pre-caches its metadata (memory hit rate > 95%).

[0131] (3) Enhance governance and data sharding: optimize strategy implementation.

[0132] In scenarios involving large table queries across different vendors, the governance module analyzes historical logs:

[0133] Identify region='Beijing' as a hotspot condition;

[0134] Output sharding recommendation: shard by region + prioritize caching the Beijing partition;

[0135] Avoid hotspot issues: Dynamically allocate shards based on node load.

[0136] The above methods enable unified aggregation of cross-vendor metadata, effectively integrating the differences in metadata formats and semantics across different vendors. This achieves bidirectional optimization of query paths and metadata.

[0137] Optionally, the method further includes:

[0138] Monitor the operational information of the first storage node;

[0139] Based on the operational information, the performance metrics of the first storage node are updated in the global metadata directory.

[0140] Monitor the operational information of the first storage node, including resource usage, execution time, and running speed.

[0141] For example, collect the resource usage (Central Processing Unit (CPU) / memory) of the first storage node and record the execution time of the sharding.

[0142] Update the storage node load status of the global directory based on the running information, and trigger the adapter to adjust the metadata collection frequency (e.g., reduce the collection frequency of high-load nodes).

[0143] Based on the above embodiments, an interaction process between Unified DNS Smart Routing and Gravitino metadata is as follows:

[0144] User request for access:

[0145] When a user submits a query, the request is accessed through a unified DNS entry point.

[0146] Unified DNS intelligent routing performs metadata queries and dynamic routing decisions.

[0147] Specifically, the unified DNS requests metadata for prod_db.table1 from Gravitino to obtain its physical storage location (such as the HDFS path of vendor A), data distribution, and current load status.

[0148] If vendor A's storage load is below a threshold (e.g., CPU utilization <70%), the route is routed to vendor A's DNS service.

[0149] If vendor A is unreachable or overloaded, automatically switch to vendor B and update the Gravitino metadata status.

[0150] Traditional DNS routing is typically based on static configurations (such as Internet Protocol (IP) lists) and cannot perceive data distribution and storage load. By deeply binding routing policies with metadata, such as routing to the nearest Beijing region storage based on table partition location (e.g., region='Beijing'), and combining real-time performance data (such as latency and throughput), dynamic weight allocation can be achieved (e.g., demoting vendors with high loads).

[0151] Metadata governance tools (such as Atlas) typically passively synchronize data, lacking cross-vendor collaboration capabilities. This application employs an active synchronization mechanism: Gravitino monitors vendor storage events (such as HDFS file creation) and updates the global directory in real time. Furthermore, it uses an AI-enhanced strategy to predict hot data based on historical query patterns and preloads metadata into an in-memory hash table, achieving the following:

[0152] Provides data partitioning information for Spark to optimize predicate pushdown.

[0153] Synchronize real-time performance metrics of the vendor's storage (such as IO latency and replica health) for unified DNS weight adjustment.

[0154] Unified DNS utilizes Gravitino metadata to achieve top-down optimized routing decisions; from the bottom up, storage layer performance data is fed back to Gravitino, triggering adjustments to metadata governance policies (such as cache refresh and route weight updates). The specific methods are as follows:

[0155] 1) Performance indicator reporting:

[0156] Each vendor's storage nodes periodically report performance data (such as QPS and latency) to Gravitino.

[0157] Gravitino analyzes the data and generates optimization suggestions (such as "Vendor B storage Asia partition has lower latency").

[0158] 2) Dynamic strategy adjustment:

[0159] Based on Gravitino's recommendations, the unified DNS dynamically updates routing policies (such as increasing vendor B's weight to 80%).

[0160] If a vendor's storage continues to be overloaded, an alarm will be triggered and an upgrade will be recommended.

[0161] 3) Locality-aware scheduling:

[0162] If the data is stored in vendor A's local data center, Spark tasks will be prioritized for scheduling to compute nodes in the same region to reduce network transmission.

[0163] Furthermore, the cross-vendor Spark query optimization layer enables federated queries across storage systems, maximizing the utilization of metadata to optimize execution plans. Specifically, this includes:

[0164] Cross-vendor Join optimization module: Collects row count and data distribution information from each vendor's table, estimates the cost of the join operation based on statistical information, and selects the optimal strategy. Specifically, the module uses the Broadcast Join strategy for small tables (<1GB) and the Shuffle Join strategy for large tables.

[0165] Distributed computing strategy module: It adopts predicate pushdown technology to push down filtering conditions to the storage layer to reduce data transmission volume; at the same time, it adopts a locality-aware scheduling strategy to prioritize scheduling computing tasks to the physical location of the storage node.

[0166] The result aggregation optimization module performs partial aggregation at the storage layer, reducing the amount of intermediate data transfer, and caches frequently accessed aggregation results in Gravitino to accelerate subsequent queries.

[0167] Virtual path mapping module: Utilizing Gravitino's virtual path mapping technology, this module enables transparent access to physical storage locations, masking differences in physical storage. Specifically, it establishes virtual path mapping relationships, allowing Spark to access data via the `gravitino: / / ` protocol, which Gravitino automatically converts to physical paths.

[0168] Performance feedback mechanism module: The storage layer reports metrics to the unified DNS in real time, triggering dynamic adjustments to the routing policy.

[0169] By using the methods described above, queries can be optimized and query efficiency improved.

[0170] As a specific example, such as Figure 2 As shown, the data query system includes a unified DNS intelligent routing layer, a Gravitino metadata federated governance layer, and a cross-vendor Spark query optimization layer.

[0171] The unified DNS intelligent routing layer includes:

[0172] Hostname resolution module: used to resolve logical identifiers in user requests into storage service addresses specific to the vendor;

[0173] Dynamic routing strategy module: This module dynamically adjusts the distribution path based on metadata and real-time performance metrics provided by Gravitino. It uses a Q-Learning model to train the optimal distribution path and dynamically adjusts DNS weights based on the real-time performance of storage nodes (such as latency and throughput).

[0174] Fault self-healing and load balancing module: Used for primary / standby failover and weight allocation to ensure high system availability. When a vendor becomes unavailable, it automatically switches to the standby vendor and updates the Gravitino metadata status.

[0175] The Hostname resolution module, dynamic routing strategy module, and fault self-healing and load balancing module work closely together to achieve intelligent request distribution, load balancing, and fault recovery. These three modules form a closed-loop processing chain: the Hostname resolution module acts as the entry point, processing user request identifiers; the dynamic routing strategy module optimizes path decisions based on real-time data; and the fault self-healing and load balancing module performs fault tolerance and resource allocation. Their interaction is event-driven and data-sharing-based, ensuring the system's high efficiency, stability, and real-time responsiveness.

[0176] The three modules are interconnected in a serial dependency: the output of the Hostname resolution module directly drives the input of the dynamic routing strategy module; the decision result of the dynamic routing strategy module is output to the fault self-healing and load balancing module for execution; the fault self-healing module influences the weight calculation of the dynamic routing strategy through event feedback, forming a closed loop within the layer. The three modules establish direct connections through data flow and event mechanisms, forming a collaborative architecture of "resolution-decision-execution".

[0177] Hostname Resolution Module → Dynamic Routing Policy Module: The output of the Hostname Resolution Module (the resolved physical address or logical identifier) ​​serves as the input to the Dynamic Routing Policy Module. For example, the resolved vendor physical address (such as hdfs: / / vendorA / path) is passed to the Dynamic Routing Policy Module for path optimization decisions. This connection is a unidirectional data flow, ensuring that routing decisions are based on accurate destination addresses.

[0178] Dynamic Routing Strategy Module → Fault Healing and Load Balancing Module: The output of the Dynamic Routing Strategy Module (dynamic weights and routing rules) serves as the input to the Fault Healing and Load Balancing Module. For example, the weight allocation calculated based on node scores (e.g., vendor A with 70% weight, vendor B with 30% weight) is input to the load balancing module for actual request distribution. Simultaneously, routing decision events (such as path switching) trigger the execution of the Fault Healing Module.

[0179] Fault Healing and Load Balancing Module → Dynamic Routing Strategy Module: The output of the fault healing module (fault events and health status) serves as feedback input to the dynamic routing strategy module. For example, when vendor unavailability is detected (e.g., three consecutive failed health checks), the fault healing module generates an event notification (e.g., weight set to 0), triggering the dynamic routing strategy module to recalculate the weights. Furthermore, load balancing status (e.g., weight adjustment records) influences the dynamic routing scoring model through internal shared data (e.g., sliding window cache).

[0180] The three modules have direct data channels with the Gravitino layer, forming a metadata-driven decision-making closed loop, as detailed below:

[0181] Hostname resolution module and Gravitino layer: The resolved logical identifier is used as input to trigger the dynamic routing module to initiate a metadata query to Gravitino;

[0182] The dynamic routing strategy module and the Gravitino layer: each time an access request is made, a query is sent to the Gravitino metadata aggregation module;

[0183] Fault self-healing module and Gravitino layer: Data reporting, including primary / standby switchover events (e.g., vendor A → vendor B) and node health status changes (e.g., replica health < 90%).

[0184] The interaction with the cross-vendor Spark query optimization layer is mainly manifested as a closed-loop feedback of performance data, with both direct and indirect channels:

[0185] Direct performance feedback (from the Spark layer to the fault self-healing module), and real-time direct reporting of storage layer performance metrics;

[0186] Indirect performance optimization (unifying DNS to the Spark layer), impact of routing decisions: the DNS response returns the target vendor address, and Spark accesses physical storage based on this address.

[0187] The Gravitino metadata federated governance layer is used to achieve unified management, governance, and optimization of metadata across vendors. Specifically, it includes:

[0188] 1) Metadata Aggregation Module: This module integrates metadata from multiple vendors' storage systems via adapters to build a global directory. It uses a distributed database to store the global directory, supporting high-concurrency access.

[0189] 2) Dynamic Synchronization Module: Monitors metadata changes from various vendors in real time and automatically updates the global view. It employs an event-driven synchronization mechanism to ensure the real-time performance of the global directory.

[0190] 3) AI-Enhanced Governance Module: Based on historical query logs and machine learning models, this module pre-caches hotspot metadata and optimizes data lineage tracing. It constructs a data lineage graph to support rapid source tracing.

[0191] 4) Data sharding and scheduling module: Dynamically shards data according to query characteristics, executes query tasks in parallel, and achieves load balancing.

[0192] A cross-vendor Spark query optimization layer is used to maximize the use of metadata to optimize the execution plan. Specifically, it includes:

[0193] 1) Cross-vendor Join optimization module: Collects row count and data distribution information from each vendor's table, estimates the cost of the join operation based on statistical information, and selects the optimal strategy;

[0194] 2) Distributed computing strategy module: It adopts predicate pushdown technology to push down filtering conditions to the storage layer to reduce data transmission volume; at the same time, it adopts a locality-aware scheduling strategy to prioritize scheduling computing tasks to the physical location of the storage node;

[0195] 3) Result aggregation optimization module: Performs partial aggregation in the storage layer to reduce the amount of intermediate data transfer, and caches frequently accessed aggregation results in Gravitino to accelerate subsequent queries;

[0196] 4) Virtual path mapping module: Through Gravitino's virtual path mapping technology, transparent access to physical storage location is achieved, shielding differences in physical storage;

[0197] 5) Performance feedback mechanism module: The storage layer reports metrics to the unified DNS in real time, triggering dynamic adjustments to the routing policy.

[0198] The five modules are connected through chain dependencies and cross-feedback to form a layered collaborative architecture, specifically including:

[0199] 1. Virtual path mapping to Join optimization / computation strategy module (basic support)

[0200] Interaction mechanism: All queries must first convert the logical address through the virtual path mapping module (e.g., gravitino: / / prod_db.table1 → hdfs: / / vendorA / table1).

[0201] Data flow: The physical path serves as the input to the Join optimization module (used to locate the data source) and is also used by the computation strategy module for local scheduling.

[0202] 2. From the Join optimization module to the distributed computing strategy module (execution plan transfer)

[0203] Interaction Mechanism: The execution plan generated by the Join optimization module (such as "BroadcastJoin on table X") directly drives the task splitting logic of the computation strategy module.

[0204] Data dependency: The Join strategy depends on the physical location of the output of the virtual path mapping (e.g., Broadcast Join is only enabled when the smaller table is located in a low-latency vendor).

[0205] 3. From the distributed computing strategy module to the result aggregation and optimization module (intermediate result transfer)

[0206] Data flow: The distributed task output of the computation strategy module (such as filtered data shards) is input to the result aggregation module for pre-aggregation.

[0207] 4. From the results aggregation and optimization module to the performance feedback mechanism module (indicator collection entry point)

[0208] Interaction mechanism: During execution, the aggregation module records key metrics (such as aggregation time and network transmission volume) and pushes them to the performance feedback module.

[0209] 5. Performance feedback mechanism module to virtual path mapping module (closed-loop effect)

[0210] Interaction mechanism: The vendor performance data reported by the feedback module (such as a spike in vendor B's latency) triggers a unified DNS weight adjustment → Subsequent queries will be automatically routed to the high-weight vendor when mapped via virtual path.

[0211] This system deeply integrates DNS routing capabilities with the Gravitino metadata governance tool to achieve bidirectional optimization of query paths and metadata, thereby improving the overall performance and availability of the distributed storage system.

[0212] Based on the above distributed storage architecture, the data query method includes the following steps:

[0213] Step 1: Initialize DNS weights.

[0214] Step 2: User request access and unified DNS routing.

[0215] (1) Request Identifier Resolution: When a user submits a query, he or she carries a logical identifier, and the unified DNS resolves this identifier as a virtual path.

[0216] (2) Intelligent distribution strategy: Generate routing rules based on metadata provided by Gravitino (such as data distribution and vendor load). Train the optimal distribution path through the Q-Learning model.

[0217] Step 3: Calculate the storage node score.

[0218] The collected performance metrics data are normalized and transformed into a unified node score. Specifically, the following scoring rules are adopted:

[0219] Multi-dimensional performance metrics are converted into a unified node score (0-100 points) for weight adjustment.

[0220] Step 3-1: Indicator Normalization:

[0221] Each metric is normalized (e.g., lower latency results in a higher score, higher throughput results in a higher score). For example, the latency scoring formula is as follows:

[0222] =max(0, 100− ×100)

[0223] in, This represents the lower limit of the ideal delay (e.g., 1ms corresponds to 100 points). This indicates the maximum tolerance delay (e.g., 20ms corresponds to 0 points).

[0224] Taking the CPU scoring formula as an example: =100 − CPU utilization

[0225] Step 3-2: Weight Allocation:

[0226] Assign weights to different metrics based on business needs (e.g., latency 40%, throughput 30%, CPU 30%). Perform a comprehensive score, for example, using the following formula:

[0227] =0.4× 0.3× +0.3×

[0228] Step 3-3: Dynamic Score Update:

[0229] The node's overall score is calculated every 5 seconds and updated to the unified DNS policy engine.

[0230] Step 4: Dynamic weight allocation

[0231] The DNS weight is dynamically adjusted based on the real-time performance of the storage nodes (e.g., latency <10ms, replica health >95%), prioritizing routing to the optimal provider. The adjustment method is as follows:

[0232] (1) Real-time performance data acquisition

[0233] 1) Deploy the monitoring agent.

[0234] Deploy a lightweight monitoring agent on each storage node to periodically (e.g., every second) collect performance metrics, including:

[0235] Latency: Storage node response time (such as Ping latency, Application Programming Interface (API) request latency);

[0236] Throughput: Requests processed per second (Queries Per Second, QPS), network bandwidth utilization;

[0237] Resource utilization: Central Processing Unit (CPU), memory, and disk input / output operations per second (IOPS).

[0238] Health status: number of replicas, node liveness status (e.g., HTTP health check interface / health).

[0239] 2) Data reporting and aggregation

[0240] The monitoring agent reports data to a central monitoring system (such as Prometheus or InfluxDB).

[0241] Data storage: Historical performance data is recorded using a time-series database, supporting trend analysis.

[0242] (2) Dynamic weight adjustment

[0243] DNS weights are dynamically adjusted based on node ratings, prioritizing routes to nodes with higher ratings. Implementation methods include:

[0244] 1) Establish weight mapping rules:

[0245] Node weight (i.e., first weight) is directly proportional to its score; the higher the score, the greater the weight. The formula is:

[0246] Where Wi is the weight of node i (i.e., the first weight), Sj represents the performance score of the j-th node, Si is the performance score of the current node i, and N is the number of available nodes.

[0247]

[0248] 2) Smooth transition strategy:

[0249] Use a sliding window (such as the average of the past 3 ratings) to avoid drastic fluctuations in weight; set a minimum weight (such as 5%) to prevent nodes from being completely removed.

[0250] 3) Failover and weight reduction:

[0251] Health check failure: If a node fails three consecutive health checks, its first weight is set to 0, and traffic is switched to the backup node.

[0252] Performance degradation: If the performance score is lower than the threshold (e.g., 60 points), the first weight will decrease proportionally (e.g., for every 10 points the score drops, the weight will decrease by 20%).

[0253] (3) Dynamic configuration update of unified DNS

[0254] The adjusted weights are applied to the DNS service in real time, and user requests are distributed according to the real-time weights. The configuration center obtains the current node weight list.

[0255] Select the target node according to the weight ratio (e.g., storage nodes A and B with a weight of 70:30, 70% of requests are routed to storage node A).

[0256] Performs self-healing: If the primary provider's DNS is unreachable (e.g., timeouts occur 3 times), automatically switches to the backup provider and updates the Gravitino metadata status.

[0257] Step 5: Gravitino Metadata Federation and Optimization

[0258] The specific processing steps of the metadata aggregation module in the Gravitino metadata federation governance layer are as follows:

[0259] (1) Metadata aggregation

[0260] 1) Collect storage metadata (such as table structure and partition information) from various vendors through the Gravitino agent, and build a global metadata catalog, including:

[0261] A logical view that stores all metadata, including table name, column information, partition key, storage location, etc.

[0262] Use a distributed database (such as Apache HBase or Cassandra) to store a global directory of metadata, supporting high-concurrency access.

[0263] Metadata association: Associate the same logical table in different storage systems through a unique identifier (such as a Universally Unique Identifier (UUID)).

[0264] 2) Dynamic metadata synchronization

[0265] Event-driven synchronization: Listen for metadata change events in the storage system (such as HDFS's notification interface (INotify) and S3's event notifications); trigger Gravitino's metadata update process to ensure the real-time nature of the global metadata directory.

[0266] Incremental synchronization: Only the changed parts are synchronized (such as adding partitions or modifying table structures), reducing network overhead.

[0267] (2) Update global metadata

[0268] Based on the collected metadata information, update the global metadata. Specifically, the following update strategy is adopted:

[0269] Add a new table or partition: Adds a new metadata record to the global metadata directory;

[0270] Data deletion: When data deletion is detected, update the data status in the global metadata directory;

[0271] Data modification: When data modification is detected, update the data version information in the global metadata directory.

[0272] (3) Establishing data lineage

[0273] Constructing a data lineage graph: Automatically generating cross-vendor data lineage graphs (such as vendor A table → vendor B table ETL link), tracking tables, and dependencies of data extraction, transformation, and loading tasks (Extract, Transform, Load, ETL) and downstream applications;

[0274] Use graph databases (such as Neo4j) to store lineage relationships to support rapid tracing.

[0275] (4) Perform data preheating

[0276] Based on historical query patterns, time series analysis (such as ARIMA) models are used to predict future hot tables, and metadata is pre-warmed into an in-memory cache. Specifically, based on the predicted hot tables, data is pre-loaded into an in-memory hash table in the background via a Spark task to achieve fast response.

[0277] Step 6: Query optimization across vendor-specific distributed computing frameworks (Spark)

[0278] In federated queries across storage systems, maximize the use of metadata to optimize the execution plan. The specific implementation steps for cross-vendor join optimization in the cross-vendor Spark query optimization layer are as follows:

[0279] (1) Estimate the cost of Join

[0280] Collect the number of rows and data distribution (such as maximum and minimum values) from each vendor's table into Gravitino. Estimate the cost of the Join operation based on the statistical information (such as Shuffle Join vs. Broadcast Join) and select the optimal strategy.

[0281] Example: Use Broadcast Join for small tables (<1GB) and Shuffle Join for large tables.

[0282] (2) Perform data sharding

[0283] Based on the estimated join cost, the large table is sharded according to the partitioning rules provided by Gravitino, and Shuffle Join is performed in parallel:

[0284] 1) Sharding by metadata partition key: Split the query task into parallel subtasks by partition key (such as date).

[0285] 2) Load balancing: Dynamically allocate shards based on the load of storage nodes to avoid hotspot issues.

[0286] (3) Perform distributed computing

[0287] 1) Execute a Spark task, which includes the following steps:

[0288] Reading data: Access the physical storage address to retrieve data through Gravitino's virtual path mapping mechanism;

[0289] Perform Join processing: Execute data association according to the pre-selected Join strategy;

[0290] Perform filtering and aggregation: Filter and aggregate based on query conditions;

[0291] Write back the result: Store the processed result in the target storage system.

[0292] 2) Optimize the query results, specifically including:

[0293] Predicate pushdown: Push filtering conditions down to the storage layer to reduce data transfer (e.g., only return `WHERE region='Beijing'` data).

[0294] Locality-aware scheduling: Prioritize scheduling computing tasks to the physical location of storage nodes to reduce network overhead.

[0295] Distributed aggregation: Perform partial aggregation (such as summation function (SUM) and counting function (COUNT)) at the storage layer to reduce the amount of intermediate data transmission.

[0296] Cache intermediate results: Cache frequently accessed aggregation results (such as daily statistical reports) in Gravitino to speed up subsequent queries.

[0297] Step 7: Data returned to user

[0298] Aggregate result sets from multiple vendors to achieve a transparent data access and feedback loop.

[0299] (1) Establish virtual path mapping:

[0300] Spark accesses data via the `gravitino: / / ` protocol, which Gravitino automatically translates into physical paths (e.g., `hdfs: / / vendorA / path`). Path redirection is supported (e.g., automatically updating mappings during storage migrations).

[0301] (2) Establish a performance feedback mechanism:

[0302] The storage layer reports metrics (such as Queries Per Second (QPS) and CPU utilization) to the unified DNS in real time, triggering dynamic adjustments to routing policies.

[0303] The above process can be found in [reference]. Figure 3 As shown.

[0304] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application, as shown below. Figure 4 As shown, the data query device 400 includes:

[0305] The receiving module 401 is used to receive a first request from a user, the first request being used to query data from the storage node;

[0306] The first determining module 402 is configured to, in response to the first request, determine at least two storage nodes corresponding to the first request, wherein the at least two storage nodes include heterogeneous storage nodes distributed in different regions.

[0307] The second determining module 403 is used to determine the first weight of the at least two storage nodes respectively, wherein the first weight is determined based on the performance indicators of the at least two storage nodes;

[0308] The distribution module 404 is configured to distribute the first request to a first storage node according to the first weight, wherein the at least two storage nodes include the first storage node.

[0309] Optionally, the first determining module 402 is specifically used for:

[0310] Based on the first request, obtain the network address information of at least two storage nodes;

[0311] Based on a pre-built global metadata directory, the at least two storage nodes are determined according to the network address information. The global metadata directory includes a mapping relationship between storage nodes and network address information.

[0312] Optionally, the second determining module 403 includes:

[0313] The acquisition submodule is used to acquire the performance metrics of the at least two storage nodes based on a pre-built global metadata directory;

[0314] The first determining submodule is used to determine the performance score of the at least two storage nodes based on the performance indicators of the at least two storage nodes, wherein the performance score is a score of the operating performance of the storage nodes;

[0315] The second determining submodule is used to determine the first weight of the at least two storage nodes based on the performance scores of the at least two storage nodes.

[0316] Optionally, the second determining submodule is specifically used to perform any of the following:

[0317] Based on the preset correspondence between the performance score and the weight, a first weight corresponding to the performance score of each of the at least two storage nodes is determined;

[0318] A second weight for each storage node is determined based on the proportion of its performance score to the total performance score of the at least two storage nodes. A first weight for each storage node is determined based on its historical weight and the second weight, wherein the historical weight is the historical first weight of each storage node.

[0319] Optionally, the device further includes at least one of the following:

[0320] The first setting module is used to set the first weight of the first storage node to 0 if the first storage node fails to perform health checks M times, and to switch the storage node corresponding to the first request from the first storage node to the second storage node among the at least two storage nodes according to the magnitude of the first weight, where M is an integer greater than 0.

[0321] The second setting module is used to set the weight value of the second weight of the first storage node to a third weight value when the second weight of the first storage node is less than a preset value.

[0322] Optionally, the device further includes:

[0323] A construction module is used to obtain the metadata of the at least two storage nodes and construct a global metadata directory based on the metadata, wherein the metadata includes at least one of the data distribution information and performance information of the at least two nodes;

[0324] A listening module is used to listen for metadata events and update the metadata in the global metadata directory based on the metadata events. The metadata events include at least one of metadata change events and addition events.

[0325] Optionally, the device further includes:

[0326] The monitoring module is used to monitor the operating information of the first storage node;

[0327] The update module is used to update the performance metrics of the first storage node in the global metadata directory based on the operational information.

[0328] The data query device can achieve Figure 1 The various processes implemented in the method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0329] It should be noted that the electronic device provided in this application embodiment is capable of executing the above-described data query device. Therefore, all implementation methods in the above-described data query method embodiments are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0330] See Figure 5 As shown in the figure, this application embodiment also provides an electronic device, including a bus 601, a transceiver 602, an antenna 603, a bus interface 604, a processor 605, and a memory 606.

[0331] Processor 605, used for:

[0332] Receive the user's first request, which is used to query data on the storage node;

[0333] In response to the first request, at least two storage nodes corresponding to the first request are identified, the at least two storage nodes including heterogeneous storage nodes distributed in different regions;

[0334] A first weight is determined for each of the at least two storage nodes, and the first weight is determined based on the performance indicators of the at least two storage nodes;

[0335] According to the first weight, the first request is distributed to the first storage node, wherein the at least two storage nodes include the first storage node.

[0336] exist Figure 5 In this document, a bus architecture (represented by bus 601) is used. Bus 601 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 605 and memory represented by memory 606. Bus 601 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 604 provides an interface between bus 601 and transceiver 602. Transceiver 602 may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 605 is transmitted over a wireless medium via antenna 603, which further receives data and transmits data to processor 605.

[0337] Processor 605 manages bus 601 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 606 can be used to store data used by processor 605 during operation.

[0338] Alternatively, the processor 605 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0339] Optionally, the processor 605 executes the determination of at least two storage nodes corresponding to the first request, including:

[0340] Based on the first request, obtain the network address information of at least two storage nodes;

[0341] Based on a pre-built global metadata directory, the at least two storage nodes are determined according to the network address information. The global metadata directory includes a mapping relationship between storage nodes and network address information.

[0342] Optionally, the processor 605 performs the process of determining the first weights of the at least two storage nodes, including:

[0343] Based on a pre-built global metadata directory, obtain the performance metrics of the at least two storage nodes;

[0344] The performance scores of the at least two storage nodes are determined based on their respective performance metrics, and the performance scores are scores of the operating performance of the storage nodes.

[0345] The first weight of the at least two storage nodes is determined based on their performance scores.

[0346] Optionally, the processor 605 performs the process of determining the first weight of the at least two storage nodes based on their performance scores, including any of the following:

[0347] Based on the preset correspondence between the performance score and the weight, a first weight corresponding to the performance score of each of the at least two storage nodes is determined;

[0348] A second weight for each storage node is determined based on the proportion of its performance score to the total performance score of the at least two storage nodes. A first weight for each storage node is determined based on its historical weight and the second weight, wherein the historical weight is the historical first weight of each storage node.

[0349] Optionally, the processor 605 is also configured to perform at least one of the following:

[0350] If the first storage node fails to perform health checks M times, the first weight of the first storage node is set to 0, and the storage node corresponding to the first request is switched from the first storage node to the second storage node among the at least two storage nodes according to the magnitude of the first weight, where M is an integer greater than 0.

[0351] If the second weight of the first storage node is less than a preset value, the weight value of the second weight of the first storage node is set to the third weight value.

[0352] Optionally, the processor 605 is also configured to perform at least one of the following:

[0353] Obtain metadata from the at least two storage nodes and construct a global metadata directory based on the metadata, wherein the metadata includes at least one of the data distribution information and performance information of the at least two nodes;

[0354] Listen for metadata events and update the metadata in the global metadata directory based on the metadata events, wherein the metadata events include at least one of metadata change events and addition events.

[0355] Optionally, the processor 605 is also configured to perform at least one of the following:

[0356] Monitor the operational information of the first storage node;

[0357] Based on the operational information, the performance metrics of the first storage node are updated in the global metadata directory.

[0358] It should be noted that the electronic device provided in this application embodiment is a device capable of executing the above-described data query method. Therefore, all implementation methods in the above-described data query method embodiments are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0359] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described data query method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0360] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0361] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0362] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0363] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data query method, characterized in that, include: Receive the user's first request, which is used to query data on the storage node; In response to the first request, at least two storage nodes corresponding to the first request are identified, the at least two storage nodes including heterogeneous storage nodes distributed in different regions; A first weight is determined for each of the at least two storage nodes, and the first weight is determined based on the performance indicators of the at least two storage nodes; According to the first weight, the first request is distributed to the first storage node, wherein the at least two storage nodes include the first storage node.

2. The method according to claim 1, characterized in that, The determination of at least two storage nodes corresponding to the first request includes: Based on the first request, obtain the network address information of at least two storage nodes; Based on a pre-built global metadata directory, the at least two storage nodes are determined according to the network address information. The global metadata directory includes a mapping relationship between storage nodes and network address information.

3. The method according to claim 1, characterized in that, Determining the first weights of the at least two storage nodes includes: Based on a pre-built global metadata directory, obtain the performance metrics of the at least two storage nodes; The performance scores of the at least two storage nodes are determined based on their respective performance metrics, and the performance scores are scores of the operating performance of the storage nodes. The first weight of the at least two storage nodes is determined based on their performance scores.

4. The method according to claim 3, characterized in that, The determination of the first weight of the at least two storage nodes based on their performance scores includes any one of the following: Based on the preset correspondence between the performance score and the weight, a first weight corresponding to the performance score of each of the at least two storage nodes is determined; The second weight of each storage node is determined based on the proportion of the performance score of each storage node to the total performance score of the at least two storage nodes. Based on the historical weight of each storage node and the second weight, a first weight corresponding to each storage node is determined, wherein the historical weight is the historical first weight of each storage node.

5. The method according to claim 4, characterized in that, The method further includes at least one of the following: If the first storage node fails to perform health checks M times, the first weight of the first storage node is set to 0, and the storage node corresponding to the first request is switched from the first storage node to the second storage node among the at least two storage nodes according to the magnitude of the first weight, where M is an integer greater than 0. If the second weight of the first storage node is less than a preset value, the weight value of the second weight of the first storage node is set to the third weight value.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain metadata from the at least two storage nodes and construct a global metadata directory based on the metadata, wherein the metadata includes at least one of the data distribution information and performance information of the at least two nodes; Listen for metadata events and update the metadata in the global metadata directory based on the metadata events, wherein the metadata events include at least one of metadata change events and addition events.

7. The method according to claim 6, characterized in that, The method further includes: Monitor the operational information of the first storage node; Based on the operational information, the performance metrics of the first storage node are updated in the global metadata directory.

8. A data query device, characterized in that, include: The receiving module is used to receive the user's first request, which is used to query the data of the storage node. A first determining module is configured to, in response to the first request, determine at least two storage nodes corresponding to the first request, wherein the at least two storage nodes include heterogeneous storage nodes distributed in different regions; The second determining module is used to determine the first weight of each of the at least two storage nodes, wherein the first weight is determined based on the performance indicators of the at least two storage nodes; A distribution module is configured to distribute the first request to a first storage node according to the first weight, wherein the at least two storage nodes include the first storage node.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data query method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data query method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data query method as described in any one of claims 1 to 7.

Images