A system and method for dynamic switching of data source registration

Abstract

The present invention relates to the technical field of electronic digital data processing, and in particular to a system and method for dynamic data source registration and switching. The method comprises the following steps: step 1, data storage; step 2, data call; step 2.1, service request parsing and request feature extraction, extracting key information from the service request to provide a basis for subsequent data source matching; determining the weight range and performance indicator requirements of the required data source based on the urgency and performance requirements of the service request; in this solution, dynamic registration, clustering, and real-time scheduling of data sources are achieved through a hierarchical aggregation model (small collection → large collection) and an intelligent matching engine, and the stability and efficiency of data access are guaranteed through dynamic weight adjustment and an abnormality fuse mechanism, thereby solving the core problems of traditional solutions in terms of flexibility, intelligence, and reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a system and method for dynamic switching of data source registration. Background Art

[0002] Data source registration dynamic switching is a technology that dynamically manages multiple data source connections and flexibly switches data access paths during application operation;

[0003] The traditional switching method solidifies the data source configuration at the system startup phase, making it difficult to cope with the dynamic addition, deletion, or routing adjustments of data sources due to business changes. For example, in multi-tenant scenarios, independent data sources cannot be allocated to new tenants in real time. There is a lack of automatic clustering and matching mechanisms based on data features (such as business tags and performance indicators), and manual coding is required to implement data source routing, resulting in low call efficiency. It is also impossible to dynamically optimize the path based on real-time performance data, which can easily lead to high latency or resource waste.

[0004] To this end, the present invention proposes a system and method for dynamic switching of data source registration. Through a hierarchical aggregation model (small collection → large collection) and an intelligent matching engine, dynamic registration, clustering, and real-time scheduling of data sources are achieved. Dynamic weight adjustment and an exception circuit breaker mechanism are used to ensure the stability and efficiency of data access, solving the core problems of traditional solutions in terms of flexibility, intelligence, and reliability. Summary of the Invention

[0005] Technical problem solved: Traditional switching methods are unable to cope with the dynamic addition and deletion of data sources or routing adjustments required by business changes.

[0006] In view of the deficiencies of the prior art, the present invention provides a system and method for dynamic switching of data source registration, thereby solving the technical problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A method for dynamic switching of data source registration, comprising the following steps:

[0009] Step 1, data storage step;

[0010] Step 1.1: Register small collections and build metadata. Register each data source as an independent small collection and assign a unique identifier to each data source small collection. This identifier is used for accurate identification during subsequent data storage, retrieval, and management, ensuring that there is no confusion between different data sources.

[0011] Step 1.2: Clustering and weighting of large collections: Cluster small collections with similar usage characteristics into large collections. Determine cluster relationships by analyzing the call frequency and call time distribution data of the data source within each time window. Use the DBSCAN density clustering algorithm to cluster data sources using the usage frequency and time distribution within the time window as feature vectors.

[0012] Step 1.3: Data storage and protection: Use a combination of Elasticsearch and PostgreSQL to store metadata. Ensure the consistency and integrity of the collection through data backup and version control.

[0013] Step 2: Data calling step;

[0014] Step 2.1: Business request parsing and feature extraction: extract key information from business requests to provide a basis for subsequent data source matching. Based on the urgency of the business request and performance requirements, determine the weight range and performance indicator requirements for the required data sources.

[0015] Step 2.2: Large collection matching: Use the rule engine to filter out qualified large collections based on request characteristics. Sorting the filtered large collections from high to low by weight, selecting high-weighted large collections for subsequent small collection location. Checking indicators such as call time and error rate of data sources within the large collection. If the call time exceeds the set threshold, the weight of the large collection is reduced or excluded.

[0016] Step 2.3: Small collection location. From the selected large collection, select small collections that meet the request requirements based on business and performance tags. Sort the selected small collections by weight from high to low, and select the small collection with the highest weight. Check the call duration, error rate, and other indicators of the small collection again. If the call exceeds the set threshold, skip the small collection and select the next small collection that meets the requirements.

[0017] Step 2.4: Data source call and feedback. Based on the located small collection, call the corresponding data source and perform data operations. Monitor the call time, error rate, and other indicators of the data source in real time, and feed the monitoring data back to the intelligent analysis module for dynamic adjustment of data source weights. The intelligent analysis module uses a sliding window algorithm and an exponentially weighted moving average method based on the monitoring data to dynamically adjust the weights of the small collection and the large collection. If an exception occurs during the call process, the automatic circuit breaker mechanism is triggered, suspending the call to the data source and switching to the backup data source. At the same time, the operation and maintenance personnel are notified to handle it.

[0018] In one possible implementation, the weight reduction or exclusion of a large collection is based on a set threshold judgment rule. If more than 30% of the data source calls in the large collection take longer than the threshold or the error rate exceeds the threshold, the weight of the large collection is reduced. The weight reduction is based on the formula , that is, the original weight is multiplied by 0.8.

[0019] In one possible implementation, a string matching algorithm is used to filter the small collection, and the service label and performance label of the request are matched one by one with the corresponding labels of the small collection.

[0020] In a possible implementation, the small collection weight sorting uses a quick sorting algorithm to sort the small collection weights.

[0021] In one possible implementation, the weights are dynamically adjusted using a sliding window algorithm and an exponentially weighted moving average method to dynamically adjust the weights of small and large collections. The time window size is set, the average call time and error rate indicators of the data source within the window are calculated, and the weights are adjusted according to the changes in the indicators. The formula for the exponentially weighted moving average method is: ,in is the current weight, is the smoothing coefficient, The value range is 0-1. is the indicator score in the current time window, is the weight at the previous moment.

[0022] In a possible implementation, a system for dynamic switching of data source registration executed in the above-mentioned method for dynamic switching of data source registration includes the following modules:

[0023] The data source registration module is responsible for data source entry, metadata annotation, and dynamic weight initialization to form an independent small collection unit;

[0024] The data aggregation and management module clusters small collections into large collections, dynamically calculates the weight of large collections, and completes data storage and protection;

[0025] The intelligent matching and calling module parses business requests, dynamically matches data source collections, executes calls, and provides feedback on monitoring data.

[0026] The visual management interface module provides a graphical interface for system operation, maintenance and configuration, lowering the operational threshold;

[0027] The system interaction process module is responsible for coordinating the automated execution of the entire process of data source registration, clustering, matching, and calling.

[0028] Beneficial effects compared with existing technologies:

[0029] 1. This solution achieves hierarchical dynamic management of data sources by registering and annotating metadata for small data sets, combining time window clustering and weight calculation to form large sets. Data sources can be flexibly aggregated and split based on business needs, improving the system's ability to organize multi-type and multi-domain data sources, facilitating rapid location and access to required data sources, and enhancing the flexibility and scalability of data management.

[0030] 2. This solution achieves intelligent matching and efficient switching of data sources during data access through processes such as business request parsing, large-collection matching, small-collection location, and dynamic weight adjustment. Dynamic weight optimization based on real-time monitoring data, combined with risk assessment and exception handling mechanisms, ensures the stability and reliability of data source access, improving data access efficiency and system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings.

[0032] Figure 1 is a flow chart of the method steps of the present invention;

[0033] Figure 2 Schematic diagram of the system framework of the present invention. DETAILED DESCRIPTION

[0034] Preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention can be implemented in various forms, so the present invention is not limited to the embodiments described below.

[0035] The technical solution in the embodiments of the present application is to solve the problems of the above-mentioned background technology, and the overall idea is as follows:

[0036] Example 1: Please refer to Figure 1 As shown, this embodiment introduces a method for dynamic switching of data source registration, including the following steps:

[0037] Step 1: Data storage step

[0038] Step 1.1: Small collection registration and metadata construction

[0039] Step 1.1.1. Data source information entry

[0040] The data source is the basic unit for data storage and call. Each data source (such as a MySQL database connection, a RESTful API interface, a file storage path, etc.) should be registered as a separate small collection. When entering, you must fully record basic connection information, such as the database URL (including IP, port, database name), username, password, driver type; API interface address, request method (GET / POST, etc.), request parameter format, response data format, etc.

[0041] Example: Register a small collection of MySQL databases and enter the following information:

[0042] Sub-collection ID generation: A unique identifier is assigned to each data source sub-collection for accurate identification during subsequent data storage, retrieval, and management, ensuring that different data sources are not confused. For example, a format similar to "db_001" can be used, where "db" indicates the database type and "001" is a sequential number.

[0043] Connection information record: Completely record the connection-related information of the data source; for database-type data sources, record the URL in detail. The URL must include the IP address, port number, and database name so that the program can accurately find the database location; also record the username and password as the identity credentials for accessing the database; and the driver type, which specifies which database driver to use to establish the connection; taking the MySQL database as an example, the URL is in the form of "jdbc:mysql: / / 192.168.1.100:3306 / user_db", where "192.168.1.100" is the IP address, "3306" is the port number, "user_db" is the database name, and "jdbc:mysql:" indicates the use of the MySQL JDBC driver protocol;

[0044] Information integration: The generated collection ID, recorded connection information, and other content are integrated together to form a complete data source collection registration information unit. This unit contains the key information required for subsequent data operations, laying the foundation for data storage and call;

[0045] Step 1.1.2: Metadata annotation

[0046] (1) Business tags: Business tags are used to identify the business area to which the data source belongs, so as to facilitate the rapid location of the data source according to business needs. When labeling, it is necessary to combine the actual business situation to ensure that the tag accurately reflects the purpose of the data source;

[0047] Example: If the data source is used to store user registration and login related data, the business tag is ; If used to store order creation and payment data, it is marked as ;

[0048] (2) Performance labels: Performance labels are based on the performance of the data source and can be determined by performance monitoring data over a period of time (such as average response time, throughput, etc.);

[0049] Example: If a data source can maintain low latency response (average response time < 100ms) under high concurrency, the performance label is If the throughput is high (number of requests processed per second > 1000), it is marked as ;

[0050] (3) Security labels: Security labels are used to mark the data security attributes of the data source, and are marked according to whether the data is encrypted for storage and transmission, whether it is desensitized, etc.

[0051] Example: If the user password in the data source is stored using SHA-256 encryption, the security tag is marked as If sensitive information such as the user's mobile phone number is anonymized in the query results, it will be marked as ;

[0052] Step 1.1.3. Dynamic Weight Initialization

[0053] Dynamic weights are used to measure the priority of data sources when calling data. The initial weight can be set based on historical usage data (such as usage frequency, stability, etc.) or initial configuration;

[0054] If the initial weight is set based on historical usage frequency, the formula ,in is the initial weight, The historical usage count of this data source. The total number of historical usages of all data sources;

[0055] Example: Assume there are three data sources with the following historical usage times: , , , then the initial weight of data source 1 , the initial weight of data source 2 , the initial weight of data source 3 ;

[0056] Step 1.2: Clustering and weight calculation of large collections

[0057] Step 1.2.1: Time window clustering. Time window clustering is based on the usage patterns of data sources over different time periods (such as days, weeks, and months). Small collections with similar usage characteristics are clustered into larger collections. Clustering relationships are determined by analyzing data such as the call frequency and call time distribution of the data source within each time window.

[0058] The DBSCAN density clustering algorithm is used to cluster the data sources using the frequency and time distribution of data sources within the time window as feature vectors; the neighborhood radius is set and the minimum number of samples If a data source is in its The number of samples in the neighborhood is greater than or equal to , then it is used as the core point and forms a cluster with other points in the neighborhood;

[0059] Example: In an e-commerce system, analyze the daily usage frequency of each data source; find that data sources A, B, and C are frequently called from 9:00 AM to 11:00 AM and from 3:00 PM to 5:00 PM on weekdays, while other data sources do not follow this pattern; set (indicates the usage frequency difference threshold), ,Data sources A, B, and C can be clustered into a “large collection of high-frequency usage on weekdays” through the DBSCAN algorithm;

[0060] Step 1.2.2: Weight aggregation. The weight of the large collection is calculated by weighting the weights of the internal small collections. Different weight coefficients can be assigned according to the importance or performance of the small collections during weighted calculation.

[0061] Using the weighted average algorithm, the formula is ,in is the weight of the large collection, For the A small collection of weights, For the The weight coefficient of a small collection ( ), is the number of small collections in the large collection;

[0062] Example: Assume that there are three small collections in the "weekday high-frequency use large collection", and their weights are , , , the weight coefficients are , , , then the weight of the large collection ;

[0063] Step 1.2.3: Resource limitation and domain locking. Based on business needs, clearly define the scope of resource use for each large collection, lock in the main domain direction, avoid the mixing of irrelevant data sources, and improve data call efficiency and accuracy.

[0064] For example, the "Financial Transaction Collection" specifies that only data sources related to financial transactions, such as transaction databases and payment interfaces, are included. When registering a new data source, if its business tag does not include "Financial Transaction" related content, it will not be allowed to be included in the collection.

[0065] Step 1.3: Data storage and protection

[0066] Step 1.3.1. Select the storage medium: Choose Elasticsearch + PostgreSQL to store metadata. Elasticsearch is used for full-text search, facilitating quick retrieval of metadata information. PostgreSQL is used to store structured metadata and supports complex queries and transaction processing.

[0067] Example: Create an index in Elasticsearch to store metadata for small and large collections. Each document contains detailed metadata for the small or large collection. Create a table in PostgreSQL to store structured metadata, such as the small collection ID, large collection ID, and tags.

[0068] Step 1.3.2: The collection protection mechanism ensures the data consistency and integrity of the collection through data backup and version control. Regularly perform snapshot backups of metadata, and restore to the backup state when data anomalies occur. Version records are kept for metadata modification operations to facilitate traceability and rollback.

[0069] Example: Use crontab to schedule a task to perform a snapshot backup of the Elasticsearch index and PostgreSQL database at 2:00 AM every day. Add a version number field to the metadata table, increment the version number each time the metadata is modified, and record the modification time and content.

[0070] Step 2: Data calling steps

[0071] Step 2.1: Business request analysis

[0072] Step 2.1.1: Request feature extraction: Extract key information from business requests, including business type (such as user query, order creation, etc.), data format (JSON, XML, etc.), permission level (normal user, administrator, etc.), request urgency (high, medium, low), etc., to provide a basis for subsequent data source matching;

[0073] Example: For a user's request to query an order, the extracted request features are: Business Type = "Order Query", Data Format = "JSON", Permission Level = "Ordinary User", Request Urgency = "Medium";

[0074] Step 2.1.2: Weight requirement analysis: Determine the weight range and performance indicator requirements of the required data sources based on the urgency and performance requirements of the business request. For example, high-urgency requests require high-weight, low-latency data sources; ordinary requests can choose relatively low-weight data sources.

[0075] Example: If the request urgency is "high", the data source weight must be no less than 0.7 and the average response time must be less than 200ms; if the request urgency is "low", the data source weight must be no less than 0.3 and the average response time must be less than 500ms.

[0076] Step 2.2: Large collection matching

[0077] Step 2.2.1: Rule filtering: Use a rule engine (such as Drools) to filter out a large set of matching requests based on request characteristics. Rules can be set based on business tags, performance tags, security tags, etc.

[0078] The rule engine uses a pattern matching algorithm to match request features with metadata tags in the collection. For example, if the request business type is "real-time transaction query," the collection with the business tag "real-time transaction" is filtered out.

[0079] Example: Set the rule "If the request business type is 'Real-time Transaction Query' and the request urgency is 'High', filter out the large collection of business tags containing 'Real-time Transaction' and performance tags containing 'Low Latency'";

[0080] 2.2.2. Weight sorting: sort the selected large collections from high to low according to weight, and give priority to large collections with high weights for subsequent small collection positioning;

[0081] Use the quick sort algorithm to sort the weights of the large collection, the time complexity is ;

[0082] Example: Assume there are three filtered large collections with weights: , , , after sorting, the order is big collection 2, big collection 1, big collection 3;

[0083] Step 2.2.3: Risk assessment: Check the call duration, error rate, and other indicators of the data sources in the large collection. If the threshold is exceeded (e.g., the call duration exceeds 500ms, the error rate exceeds 5%), the weight of the large collection will be reduced or excluded.

[0084] Set a threshold judgment rule. If more than a certain proportion (such as 30%) of the data source calls in the large collection take longer than the threshold or the error rate exceeds the threshold, the weight of the large collection will be reduced. The formula for reducing the weight can be used. (i.e., the original weight is multiplied by 0.8);

[0085] Example: If there are 5 data sources in a "large collection", and the call time of 2 of them exceeds 500ms, the excess ratio is 40%, then the weight of the large collection will be increased from Reduced to ;

[0086] Step 2.3: Small collection positioning

[0087] Step 2.3.1: Tag matching: From the selected large collection, select a small collection that meets the request requirements based on business tags and performance tags. Ensure that the tags of the small collection match the requested characteristics.

[0088] Use a string matching algorithm to match the requested service and performance tags with the corresponding tags in the small collection one by one;

[0089] Example: If the requested service tag is , the performance label is , filter out business tags containing and the performance tag contains A small collection of

[0090] Step 2.3.2: Weight screening: sort the selected small collections from high to low according to weight, and give priority to small collections with high weights;

[0091] Also use the quick sort algorithm to sort the weights of small collections;

[0092] Example: Assume there are 4 filtered small collections with weights , , , , after sorting, the order is small collection 4, small collection 2, small collection 1, small collection 3;

[0093] Step 2.3.3: Secondary risk assessment: Check the call time, error rate, and other indicators of the small collection again. If the set threshold is exceeded, skip the small collection and select the next small collection that meets the conditions;

[0094] Similar to the large collection risk assessment algorithm, a threshold judgment rule is set. If the call time or error rate of a small collection exceeds the threshold, the small collection will not be selected;

[0095] Example: If the call time for collection 1 is 600ms, which exceeds the set threshold of 500ms, collection 1 is skipped and collection 2 is selected for data source call.

[0096] Step 2.4: Data source call and feedback

[0097] Step 2.4.1: Call the data source based on the located small collection and perform data operations (such as query, insert, update, etc.);

[0098] Example: If the located collection is "db_001" (MySQL database collection), the database connection pool is used to obtain a connection and SQL statements are executed to perform data query operations;

[0099] Step 2.4.2: Monitoring and feedback: Real-time monitoring of data source call time, error rate, and other indicators, and feedback of monitoring data to the intelligent analysis module for dynamic adjustment of data source weights;

[0100] Example: Use Prometheus to monitor the call duration and error rate of a data source, collect data every 10 seconds, and visualize it using Grafana. At the same time, send the data to the intelligent analysis module.

[0101] Step 2.4.3: Dynamic weight adjustment: The intelligent analysis module uses the sliding window algorithm and exponentially weighted moving average method based on the monitoring data to dynamically adjust the weights of the small and large collections;

[0102] Set the time window size (e.g., 5 minutes), calculate the average call time, error rate, and other indicators of the data source within the window, and adjust the weights according to the changes in the indicators; the formula for the exponentially weighted moving average method is: ,in is the current weight, is the smoothing coefficient (value range is 0-1, such as 0.2), is the indicator score in the current time window, is the weight at the previous moment;

[0103] Example: If the current weight of the data source , in the new time window, due to the increase in call time, the new indicator score , , then the adjusted weight ;

[0104] Step 2.4.4: Exception handling: If an exception occurs during the call (such as connection failure or data reading error), the automatic circuit breaker mechanism is triggered, suspending the call to the data source and switching to the backup data source. At the same time, the operation and maintenance personnel are notified to handle the problem.

[0105] Example: Use Sentinel to implement an automatic circuit breaker mechanism. When a data source fails to be called three times in a row, a circuit breaker is triggered. For the next minute, the data source will not be called again, and the backup data source will be called instead. The operation and maintenance personnel will be notified via email or SMS.

[0106] Example 2: Please refer to Figure 2As shown, based on the method for dynamic switching of data source registration in Example 1, this embodiment provides a system for dynamic switching of data source registration, which aims to achieve dynamic management and intelligent switching of data sources through modular design. The system is specifically as follows:

[0107] 1. Data source registration module, responsible for data source entry, metadata annotation and dynamic weight initialization, forming an independent small collection unit;

[0108] 1.1. Information Entry Unit: Provides a visual interface or API interface, supports the entry of basic information (such as URL, user name, permissions, etc.) from multiple data sources such as databases, APIs, and file storage, and generates a unique small collection ID (such as "db_001");

[0109] 1.2. Metadata Annotation Unit: Automatically or manually annotate data sources with business tags (such as "user management"), performance tags (such as "low latency"), and security tags (such as "data encryption") based on business rules;

[0110] 1.3. Weight calculation unit: based on historical usage frequency (formula: ) or custom rules to generate initial weights for each small collection;

[0111] 2. Data aggregation and management module, which realizes clustering of small collections into large collections, dynamically calculates the weight of large collections, and completes data storage and protection;

[0112] 2.1. Clustering Engine Unit: This unit uses the DBSCAN algorithm to cluster data source usage patterns (such as call frequency and time distribution) within a time window, generating a large collection (such as a "weekday high-frequency usage large collection").

[0113] 2.2. Weight aggregation unit: through weighted average algorithm (formula: Calculate the weight of large collections and support dynamic adjustment of weight coefficients (such as giving priority to high-performance data sources);

[0114] 2.3. Resource Limitation Unit: This unit limits the scope of data sources within a large collection based on business areas (e.g., "financial transactions") and excludes irrelevant data sources from inclusion.

[0115] 2.4 Metadata Storage: Use a combination of Elasticsearch and PostgreSQL to store metadata for both small and large collections, supporting full-text search and structured queries (such as searching for data sources by tag).

[0116] 2.5. Data Protection Unit: Ensure data consistency through scheduled snapshot backups (e.g., daily at 2:00 AM) and version control (recording metadata modification history);

[0117] 3. Intelligent matching and calling module: parses business requests, dynamically matches data source collections, executes calls, and provides feedback on monitoring data.

[0118] 3.1. Request parser unit: extracts business request features (e.g., business type "order query" and urgency "high") and generates matching conditions (e.g., requiring data source weight ≥ 0.7 and tags containing "order processing").

[0119] 3.2. Rule Engine Unit: Implements rule matching based on Drools to filter large sets of matching conditions (e.g., "business tag contains 'real-time transaction' and performance tag contains 'low latency'");

[0120] 3.3. Weight Sorting and Risk Assessment Unit: Use a quick sort algorithm to sort large and small collections in descending order of weight, and eliminate abnormal data sources through threshold verification (for example, a call duration > 500ms triggers weight degradation);

[0121] 3.4. Data source connector unit: Based on the positioning results (such as the small collection "db_001"), it performs data operations through the database connection pool or HTTP client, supporting multiple types of interfaces such as MySQL and RESTful API;

[0122] 3.5. Monitoring feedback unit: collects call duration, error rate and other indicators in real time (e.g. Prometheus sampling every 10 seconds), and uses the exponentially weighted moving average method (formula: ) Dynamically adjust the data source weight and trigger an exception circuit breaker (for example, if Sentinel fails three times in a row, the call will be suspended);

[0123] 4. Visual management interface module provides a graphical interface for system operation and configuration, lowering the operational threshold;

[0124] 4.1. Data source management unit: view / edit small collection details, and manually adjust labels or weights;

[0125] 4.2. Large collection cluster monitoring unit: Visually display large collection members, weight change trends and clustering rules;

[0126] 4.3. Call log analysis unit: query the data source matching path, time consumption statistics and exception records of historical requests;

[0127] 4.4. Rule configuration unit: dynamically modify clustering rules, weight calculation formulas, and threshold parameters (e.g., adjust the "high latency" threshold to 300ms);

[0128] 5. System interaction process module

[0129] 5.1. Registration phase: Users enter data source information through the interface or API. The system generates a small collection and annotates metadata, completing initial weight calculation and storage.

[0130] 5.2. Clustering phase: The clustering engine regularly scans the usage data of small collections and generates large collections through the DBSCAN algorithm. The resource-limited unit filters out irrelevant data sources.

[0131] 5.3. Invocation phase: The business request triggers the parser to generate matching conditions. The rule engine filters the large collection, sorts the weights, and locates the high-priority small collection. The connector executes the call and feeds back monitoring data.

[0132] 5.4. Optimization phase: Based on real-time monitoring data, the system automatically adjusts the weight of small and large collections. Abnormal data sources trigger circuit breakers and notify operations and maintenance.

[0133] Finally, it should be noted that the above embodiments are merely examples for the purpose of illustrating the present invention and are not intended to limit the embodiments. Those skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. It is not necessary and impossible to provide an exhaustive list of all embodiments. However, obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for dynamic switching of data source registration, characterized in that: The method comprises the following steps: Step 1, data storage step; Step 1.1: Register small collections and build metadata. Register each data source as an independent small collection and assign a unique identifier to each data source small collection. This identifier is used for accurate identification during subsequent data storage, retrieval, and management, ensuring that there is no confusion between different data sources. Step 1.2: Clustering and weighting of large collections: Cluster small collections with similar usage characteristics into large collections. Determine cluster relationships by analyzing the call frequency and call time distribution data of the data source within each time window. Use the DBSCAN density clustering algorithm to cluster data sources using the usage frequency and time distribution within the time window as feature vectors. Step 1.3: Data storage and protection: Use a combination of Elasticsearch and PostgreSQL to store metadata. Ensure the consistency and integrity of the collection through data backup and version control. Step 2: Data calling step; Step 2.1: Business request parsing and feature extraction: extract key information from business requests to provide a basis for subsequent data source matching. Based on the urgency of the business request and performance requirements, determine the weight range and performance indicator requirements for the required data sources. Step 2.2: Large collection matching: Use the rule engine to filter out qualified large collections based on request characteristics. Sorting the filtered large collections from high to low by weight, selecting high-weighted large collections for subsequent small collection location. Check the call time and error rate indicators of the data sources in the large collection. If they exceed the set threshold, reduce the weight of the large collection or exclude it. Step 2.3: Small collection location. From the selected large collection, select small collections that meet the request requirements based on business and performance tags. Sort the selected small collections by weight from high to low, and select the small collection with the highest weight. Check the call duration and error rate indicators of the small collection again. If they exceed the set threshold, skip the small collection and select the next small collection that meets the requirements. Step 2.4: Data source call and feedback. Based on the located small collection, call the corresponding data source and perform data operations. Monitor the call time and error rate indicators of the data source in real time, and feed the monitoring data back to the intelligent analysis module for dynamic adjustment of the data source weight. The intelligent analysis module uses the sliding window algorithm and exponentially weighted moving average method based on the monitoring data to dynamically adjust the weights of the small collection and the large collection. If an exception occurs during the call process, the automatic circuit breaker mechanism is triggered, the call to the data source is suspended, and the backup data source is switched. At the same time, the operation and maintenance personnel are notified to handle it.

2. A method for dynamic switching of data source registration according to claim 1, characterized in that: The weight reduction or exclusion of a large collection is based on the set threshold judgment rules. If more than 30% of the data source calls in the large collection take longer than the threshold or the error rate exceeds the threshold, the weight of the large collection will be reduced; the weight reduction adopts the formula , that is, the original weight is multiplied by 0.

8.

3. A method for dynamic switching of data source registration according to claim 1, characterized in that: A string matching algorithm is used to filter the small collection, matching the requested business tags and performance tags with the corresponding tags in the small collection one by one.

4. A method for dynamic switching of data source registration according to claim 1, characterized in that: Small collection weight sorting uses a quick sorting algorithm to sort the small collection weights.

5. A method for dynamic switching of data source registration according to claim 1, characterized in that: Dynamic weight adjustment uses a sliding window algorithm and an exponentially weighted moving average method to dynamically adjust the weights of small and large collections. The time window size is set, and the average call time and error rate indicators of the data source within the window are calculated. The weight is adjusted according to the changes in the indicators. The formula for the exponentially weighted moving average method is: ,in is the current weight, is the smoothing coefficient, The value range is 0-1. is the indicator score in the current time window, is the weight at the previous moment.

6. A system for dynamic switching of data source registration executed in accordance with a method for dynamic switching of data source registration according to any one of claims 1 to 5, characterized in that: The system includes the following modules: The data source registration module is responsible for data source entry, metadata annotation, and dynamic weight initialization to form an independent small collection unit; The data aggregation and management module clusters small collections into large collections, dynamically calculates the weight of large collections, and completes data storage and protection; The intelligent matching and calling module parses business requests, dynamically matches data source collections, executes calls, and provides feedback on monitoring data. The visual management interface module provides a graphical interface for system operation, maintenance and configuration, lowering the operational threshold; The system interaction process module is responsible for coordinating the automated execution of the entire process of data source registration, clustering, matching, and calling.

Images