A method and system for dynamic routing and adaptive cleaning and acquisition of multi-source heterogeneous data

Abstract

This invention discloses a dynamic routing and adaptive cleaning and acquisition method and system for multi-source heterogeneous data, relating to the field of data acquisition technology for new energy power plants. The method includes: loading measurement point configuration files, parsing lifecycle attributes, dynamically dividing measurement points into streaming or batch acquisition channels, and matching corresponding physical constraint rules; periodically probing data source operation indicators, calculating scheduling weights, and dynamically adjusting task granularity and concurrent resources. Real-time data and historical aggregated data are acquired through dual channels to obtain raw acquisition information; the data is imported into a sliding time window divided by device, duplicate data is removed, valid corrected data is retained, and after cleaning, the data is batch-stored, and the scheduling strategy is iteratively optimized. This invention relies on measurement point attributes to achieve differentiated data routing, combined with an adaptive data cleaning mechanism, to accurately distinguish redundant and corrected data, effectively improving the quality of multi-source heterogeneous data and the stability of system operation in new energy power plants.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology for new energy power stations, and in particular to a method and system for dynamic routing and adaptive cleaning acquisition of multi-source heterogeneous data. Background Technology

[0002] New energy data acquisition serves as a core support for centralized monitoring of power plants and is widely used in the wind and photovoltaic sectors. With the expansion of installed capacity, related technologies, through the collaborative operation of time-series databases, big data platforms, and acquisition gateways, have constructed a multi-source heterogeneous data aggregation system. Specifically, this system covers the entire process from equipment sensing and protocol parsing to hierarchical storage, including key aspects such as real-time streaming writing, historical batch aggregation, and cross-source data synchronization, aiming to meet the dual needs of high-frequency monitoring and operational analysis.

[0003] However, existing data collection methods directly employ static routing and generic timestamp deduplication strategies without dynamically adapting to the semantics of new energy business. The lack of identification of the lifecycle attributes of measurement points forces real-time data to undergo aggregation logic, increasing latency. Furthermore, it fails to distinguish between manual data entry corrections and duplicate reporting, easily leading to inaccuracies in electricity statistics. In addition, rigid concurrency mechanisms struggle to detect backend load fluctuations and lack adaptive degradation capabilities when data source responses are abnormal, potentially causing system cascading failures and severely impacting the accuracy of monitoring data and the stability of system operation. Summary of the Invention

[0004] The main objective of this invention is to provide a method for dynamic routing and adaptive cleaning and acquisition of multi-source heterogeneous data.

[0005] Another objective of this invention is to propose a dynamic routing and adaptive cleaning and acquisition device for multi-source heterogeneous data.

[0006] The third objective of this invention is to provide an electronic device.

[0007] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a method for dynamic routing and adaptive cleaning and acquisition of multi-source heterogeneous data, comprising:

[0009] S1, load the measurement point configuration file, parse the explicitly defined lifecycle mode attributes of each measurement point, dynamically map the measurement points to be collected to the streaming acquisition channel or batch acquisition channel according to the lifecycle mode attributes, and load the physical constraint rules corresponding to the lifecycle mode attributes. S2 performs periodic probe tests on each data source to obtain real-time response time and throughput metrics, calculates the scheduling weight of each data source, and adjusts the granularity of historical data collection tasks and the allocation of concurrent resources. S3 directly acquires real-time running data through the streaming acquisition channel and acquires historical cumulative data through the batch acquisition channel according to the preset aggregation strategy and time window to obtain the raw acquisition data; S4 loads the raw collected data into a sliding time window based on device identifiers and searches for old data with overlapping time. If duplicates are found, they are discarded; if they are valid corrected data, they are marked and retained. The cleaned data is then written in batches to the target database, and the scheduling weights and task fragments for the next round are adaptively adjusted.

[0010] Optionally, load the measurement point configuration file, parse the explicitly defined lifecycle mode attributes of each measurement point, dynamically map the measurement points to be acquired to the streaming acquisition channel or batch acquisition channel according to the lifecycle mode attributes, and load the physical constraint rules corresponding to the lifecycle mode attributes, including: Read the lifecycle mode attribute explicitly defined in the measurement point configuration file. When the lifecycle mode attribute is marked as real-time, map the measurement point to be collected to the streaming acquisition channel bound to the high-concurrency memory queue, and load the physical constraint rules that force the aggregation function to be disabled and enable physical mutation detection. When the lifecycle mode attribute is identified as historical, the measurement points to be collected are mapped to the batch collection channel bound to the batch processing thread pool, and physical constraint rules containing aggregation strategies, time windows and time zone alignment rules are loaded.

[0011] Optionally, periodic probe tests are performed on each data source to obtain real-time response time and throughput metrics, calculate the scheduling weight of each data source, and adjust the granularity of historical data collection tasks and concurrent resource allocation, including: Periodically send lightweight queries to various data sources as probe tests and record the real-time response time of each data source. and throughput A sliding window is used to calculate the smoothing value to eliminate instantaneous fluctuations; Based on the real-time response time and the throughput Calculate the normalized response time score and throughput score:

[0012]

[0013] in, The minimum response time among all current data sources. This represents the maximum throughput across all current data sources. Utilize configurable weighting coefficients and Calculate the overall score:

[0014] The overall score is then normalized to a scheduling weight that sums to 1. ; According to the scheduling weight The number of collection points allocated to each data source is dynamically adjusted, assuming the total task is... Then it is assigned to the data source. The workload is: .

[0015] Optionally, real-time operational data can be directly acquired through a streaming acquisition channel, while historical cumulative data can be acquired through a batch acquisition channel based on a preset aggregation strategy and time window to obtain raw acquired data, including: If data acquisition is performed through the streaming acquisition channel, the latest data is directly retrieved from the latest write interface of the time series database, and format standardization processing is performed without applying any aggregation functions. If data collection is performed through the batch processing collection channel, a query statement containing time zone conversion logic and pre-aggregation functions is constructed according to the aggregation strategy and time window bound in the configuration file, and historical cumulative data after aggregation calculation is pulled from the big data platform or time series database as the raw collection data.

[0016] Optionally, the raw collected data is loaded into a sliding time window divided based on device identifiers, and old data with overlapping time is searched for. If duplicates are found, they are discarded; if they are valid corrected data, they are marked and retained. The cleaned data is then written in batches to the target database, and the scheduling weights and task fragments for the next round are adaptively adjusted, including: Maintain a sliding time window based on device ID slices in memory, with the window length being the maximum physical latency threshold defined in the configuration file. ; When new data Upon entry, does the search window contain a timestamp? Old data If there is old data with overlapping time periods Then calculate the rate of change of the value:

[0017] in, and These are the values ​​for the new data and the old data, respectively. and These are the timestamps for the new data and the old data, respectively. The rate of change of the numerical value With dynamically adjusted rate of change threshold If a comparison is made, If so, it is determined to be a duplicate entry and the new data is discarded. ,like The data is then deemed a valid correction and the new data is retained. Simultaneously trigger data correction alarms; if the data violates physical laws, it will be directly intercepted and an error will be reported. The cleaned data is written to the target database in batches, the time taken and success rate of this collection are recorded, and the load awareness model is updated for the next round of task sharding adjustment.

[0018] To achieve the above objectives, a second aspect of the present invention provides a multi-source heterogeneous data dynamic routing and adaptive cleaning acquisition device, comprising: The mapping module is used to load the measurement point configuration file, parse the life cycle mode attributes explicitly defined for each measurement point, dynamically map the measurement points to be collected to the streaming acquisition channel or batch acquisition channel according to the life cycle mode attributes, and load the physical constraint rules corresponding to the life cycle mode attributes. The testing module is used to perform periodic probe tests on each data source, obtain real-time response time and throughput metrics, calculate the scheduling weight of each data source, and adjust the granularity of historical data collection tasks and the allocation of concurrent resources. The acquisition module is used to directly acquire real-time running data through the streaming acquisition channel and acquire historical cumulative data through the batch acquisition channel according to the preset aggregation strategy and time window to obtain the raw acquisition data. The retrieval module loads the raw collected data into a sliding time window based on device identifiers, and searches for old data with overlapping time. If duplicates are found, they are discarded; if the data is valid and corrected, it is marked and retained. The cleaned data is then written in batches to the target database, and the scheduling weights and task fragments are adaptively adjusted for the next round.

[0019] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0020] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing a multi-source heterogeneous data dynamic routing and adaptive cleaning acquisition method as described in the first aspect embodiment.

[0021] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements a multi-source heterogeneous data dynamic routing and adaptive cleaning acquisition method as described in the first aspect embodiment.

[0022] The embodiments of the present invention have the following beneficial effects: This invention significantly reduces data acquisition latency and prevents system avalanche through dual-modal dynamic routing and load-aware scheduling, improving processing efficiency and stability in multi-source heterogeneous environments. Combined with a spatiotemporal normalization deduplication mechanism based on business semantic constraints, it effectively distinguishes between duplicate entries and effectively corrected data, solving the statistical bias problem caused by human intervention in new energy scenarios and improving data accuracy. Attached Figure Description

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of a dynamic routing and adaptive cleaning and acquisition method for multi-source heterogeneous data provided in an embodiment of the present invention; Figure 2 The system overall architecture and dynamic routing flowchart provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of configuration hot loading and dynamic adjustment provided in an embodiment of the present invention; Figure 4 A schematic diagram of load-aware adaptive sharding scheduling provided in an embodiment of the present invention; Figure 5 A flowchart of a spatiotemporal normalization deduplication algorithm based on business semantic constraints provided in an embodiment of the present invention; Figure 6 This is a structural diagram of a multi-source heterogeneous data dynamic routing and adaptive cleaning and acquisition device provided in an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] The following description, with reference to the accompanying drawings, describes a method and apparatus for dynamic routing and adaptive cleaning and acquisition of multi-source heterogeneous data according to embodiments of the present invention.

[0027] Example 1 This invention provides a method for dynamic routing and adaptive cleaning and acquisition of multi-source heterogeneous data, such as... Figure 1 As shown, the method includes the following steps: S1, load the measurement point configuration file, parse the explicitly defined lifecycle mode attributes of each measurement point, dynamically map the measurement points to be collected to the streaming acquisition channel or batch acquisition channel according to the lifecycle mode attributes, and load the physical constraint rules corresponding to the lifecycle mode attributes.

[0028] In this embodiment, S1 is the initial configuration parsing and routing initialization step of the system operation. Relying on the dual-modal dynamic configuration engine set in this application, the standardized classification management of all data collection points of the entire station is completed.

[0029] In this embodiment, the traditional flat, fixed configuration design of data acquisition systems is abandoned. Instead, a novel dual-modal configuration structure linking real-time and historical states is adopted, capable of adapting to the dual business needs of real-time high-frequency data acquisition and historical batch statistics at new energy power plants. In this embodiment, the configuration files for all measurement points within the system are uniformly written and stored in the YAML standard format. In addition to the basic parameters of each acquisition measurement point, the LifecycleMode lifecycle mode identifier is explicitly defined. This identifier serves as the core criterion for subsequent intelligent routing scheduling and channel allocation, ensuring unified classification logic and customizable configuration.

[0030] In this embodiment of the application, during the execution process, the local and remote synchronized test point configuration files are read and loaded line by line, the lifecycle mode attributes corresponding to each test point are accurately parsed, and the traffic binding processing of test points of different business types is completed.

[0031] In this embodiment, when the measurement point configuration is identified as RealTime, the system automatically binds high-concurrency memory queue resources to this type of measurement point, directly establishes a long-term connection with the latest write interface of the time-series database, and simultaneously loads the operation strategy constraints specific to the real-time scenario, forcibly disabling all aggregation calculation functions within the system, strictly reducing data transmission latency from the underlying operation logic, and stably maintaining millisecond-level data acquisition and writing performance. At the same time, this embodiment also enables a physical mutation detection mechanism, which can capture instantaneous operating condition mutation data during the operation of new energy equipment such as wind power, photovoltaics, and energy storage in real time, ensuring that abnormal operating characteristics are not lost.

[0032] In this embodiment, when the measurement point configuration is identified as Historical, the system automatically connects the measurement point to the built-in batch processing thread pool resource. Relying on the multi-threaded concurrency capability, it carries out the query and calculation tasks of a large amount of historical data. The system also loads the necessary constraint configuration parameters for this mode, including AggregationStrategy, TimeWindow (fixed statistical time window), and TimeZoneAlign (unified time zone alignment rule). Relying on the multi-dimensional constraint parameters, the system provides complete configuration support for subsequent cross-platform query, pre-aggregation calculation, and standardization of large amounts of historical data.

[0033] In this embodiment, the system has a complete configuration hot-load operation capability, which can realize online dynamic updating and iterative optimization of basic parameters of measurement points, life cycle modal attributes, and business constraint rules without stopping and restarting the entire service program.

[0034] like Figure 3 As shown in the figure, the automatic difference comparison mechanism after the configuration file is modified is fully demonstrated. Through intelligent comparison and analysis of the old and new versions of the configuration file, key contents such as changes in the mode of measurement points, modifications to operating parameters, and adjustments to constraint rules can be quickly and accurately located. The routing mapping relationship table inside the system is refreshed in real time to ensure that the data channels of various measurement points can be seamlessly switched and run stably without interruption during the dynamic adjustment of the configuration.

[0035] In this embodiment, the system distinguishes between real-time and historical measurement points by parsing the dual-modal configuration file, binds the corresponding transmission channels and constraint rules, and dynamically updates the routes based on the configuration hot-loading mechanism, thereby completing the initial setup of the system's acquisition routes and laying the foundation for obtaining real-time response time and throughput indicators in the future.

[0036] S2 performs periodic probe tests on each data source to obtain real-time response time and throughput metrics, calculates the scheduling weight of each data source, and adjusts the granularity of historical data collection tasks and the allocation of concurrent resources.

[0037] In this embodiment, step S2 mainly relies on the load-aware execution cluster set up in this application to continuously monitor the real-time operating status of various new energy data sources, and at the same time realize adaptive load scheduling control in all scenarios. It is a key core link to ensure the stable and reliable concurrent acquisition process of multi-source heterogeneous data and the balanced operation of load.

[0038] In this embodiment, the system will periodically send lightweight detection and query commands to each connected new energy data source during the device startup initialization phase and throughout the subsequent normal continuous operation. It will rely on a probe detection mechanism to collect key operational indicators of each data source in real time, including real-time response time. and throughput In this embodiment of the application, in order to effectively avoid the deviation of monitoring data caused by external interference factors such as on-site network fluctuations, instantaneous access peaks, and sudden changes in short-term resource usage, this embodiment of the application adopts a sliding window smoothing calculation method to perform mean optimization processing on the collected original monitoring indicators, effectively filtering abnormal fluctuation data and ensuring the objectivity and accuracy of the multi-data source operation status evaluation results.

[0039] In this embodiment, a multi-level quantization calculation method is used to complete the normalization processing of multiple types of operating indicators and the accurate solution of the comprehensive scheduling weight of the data source. In this embodiment, the globally optimal benchmark parameters are first uniformly determined to... As the minimum response time among all data sources, As the highest throughput among all data sources, the calculation relationship of two types of core evaluation scores is constructed based on the benchmark parameters:

[0040]

[0041] In this application embodiment, a weighting coefficient that can be configured and adjusted independently for the business scenario is introduced. and Strictly meet Given the constraints, staff can flexibly adjust the priority configuration between data real-time performance and platform processing capabilities based on the actual operational needs of the new energy power stations. For real-time monitoring scenarios, the priority can be appropriately increased. For batch statistics and offline analysis scenarios, the value can be appropriately increased. The value of is used as the basis for weighted fusion calculation to obtain the comprehensive performance score corresponding to a single data source:

[0042] In this embodiment, the comprehensive scores of all data sources are further subjected to global normalization to generate standardized scheduling weights with a sum of 1. The specific calculation relationship is as follows:

[0043] In this embodiment, the application strictly follows the obtained scheduling weights to complete the adaptive segmentation and allocation control of historical data acquisition tasks. The total task volume of all data acquisition points in the new energy power station is set to N, and the data is allocated to the [number of] data acquisition points. The actual data collection workload for each data source can be calculated using the corresponding formula:

[0044] In this embodiment, the overall scheduling logic has dynamic adaptation characteristics. Data sources that are in a high-load, high-latency running state will receive a lower scheduling weight, and the system will automatically reduce the number of task shards of such data sources. Meanwhile, data sources that are stable in operation, have a lower load, and stronger throughput will have a higher scheduling weight, and the number of corresponding collection tasks will be reasonably increased, thereby achieving natural load balancing among multiple data sources at the task allocation level.

[0045] In this embodiment, the system also incorporates a robust backpressure feedback control mechanism to control the real-time response time of any single data source. If the system exceeds the preset threshold (the default critical threshold is set to 5 seconds), the system will immediately trigger a tiered degradation protection strategy, temporarily suspending the retrieval of historical data for non-critical categories under that data source, and only retaining the real-time data collection work corresponding to the core business. This effectively avoids the occurrence of chain blocking problems in multi-source high-concurrency access scenarios and eliminates the risk of service avalanche.

[0046] In this embodiment of the application, after the operating indicators of the abnormal data source gradually recover to the normal and reasonable range, the system will gradually call back the scheduling weight parameters of the data source, and the corresponding number of collection task fragments will also be automatically restored, ensuring the continuous and stable progress of the overall collection business.

[0047] like Figure 4 As shown in the diagram, the complete closed-loop control logic of real-time probe monitoring, quantitative calculation of operating indicators, reasonable allocation of dynamic weights, adaptive sharding of acquisition tasks, and back pressure triggering degradation protection is fully and clearly presented. It clearly marks the complete data flow path and scheduling control execution logic within the load-aware execution cluster of this application, and intuitively explains the core implementation principle of dynamic resource allocation and intelligent load balancing among multiple data sources, providing reliable architectural support for the stable and collaborative operation of the overall multi-source acquisition system.

[0048] In this embodiment, probes are used to monitor the operating indicators of multiple data sources. Normalization calculation and dynamic weight allocation are combined to achieve adaptive task sharding. With the back pressure degradation mechanism, load balancing and stable scheduling of new energy multi-source acquisition are completed, laying the foundation for obtaining the raw acquisition data in the future.

[0049] S3 directly acquires real-time running data through the streaming acquisition channel and obtains historical cumulative data through the batch acquisition channel based on the preset aggregation strategy and time window to obtain the raw acquisition data.

[0050] In this embodiment, step S3 relies on the intelligent routing scheduling center set up in this application, strictly follows the generated complete routing mapping relationship, and adopts a differentiated acquisition strategy of different channels and different modes to complete the accurate acquisition of multi-source heterogeneous data of new energy power stations. The core purpose is to adapt to the different data processing needs of real-time monitoring business and historical statistics business, and ensure the timeliness and quality of data acquisition for the two types of businesses.

[0051] In this embodiment, the application utilizes the core scheduling capabilities of the intelligent routing scheduling center to achieve precise distribution of measurement point data under different lifecycle modes, ensuring that real-time data and historical data are collected according to their respective optimal paths, avoiding mutual interference during the collection of different types of data, and improving overall collection efficiency.

[0052] In this embodiment, for the streaming acquisition channel corresponding to the divided RealTime measurement points, the system automatically calls a dedicated high-speed access protocol to directly establish a dedicated connection with the real-time write interface of the time series database, without performing any form of data aggregation operation, thereby minimizing data processing overhead.

[0053] In this embodiment, the channel only performs format standardization processing and field mapping on the collected raw data packets to ensure that the data format is standardized and the fields are consistent. At the same time, it ensures millisecond-level data collection and transmission timeliness with the lowest processing latency, meeting the business needs of real-time monitoring and instantaneous operating condition capture of new energy power stations.

[0054] In this embodiment, the streaming acquisition channel is also equipped with a physical mutation detection mechanism, which can capture the sudden changes in key operating parameters of new energy power generation equipment in real time, including the instantaneous fluctuations of core parameters such as power generation, bus voltage, and unit output. It fully preserves the instantaneous operating characteristics of the equipment and provides accurate raw data support for subsequent abnormal operating condition analysis and fault tracing.

[0055] In this embodiment, for the batch acquisition channel corresponding to the divided Historical measurement points, the system automatically reads the preset constraint parameters in the configuration file, including the AggregationStrategy, the fixed TimeWindow, and the TimeZoneAlign. Based on these parameters, it automatically concatenates and generates query instructions that are compatible with different storage media, thereby achieving compatibility and adaptation across multiple storage platforms.

[0056] In this embodiment, for historical data stored in a time-series database, the system automatically generates standardized Line Protocol statements and customized query SQL to ensure the compatibility and efficiency of query commands. For massive historical data stored on a big data platform, the system constructs complex execution statements with embedded time zone conversion logic and pre-aggregation functions, and simultaneously schedules MapReduce tasks to complete batch pre-aggregation calculations of massive historical data, effectively reducing the data volume and outputting aggregated historical cumulative data that meets business needs, adapting to batch data processing scenarios such as historical statistics and offline analysis.

[0057] In this embodiment, the synchronous acquisition of real-time data and historical data is completed through the coordinated operation of streaming acquisition channels and batch acquisition channels. The two types of data with different attributes and purposes are summarized and integrated to form a complete multi-source heterogeneous original acquisition dataset, which lays the foundation for subsequent retrieval of whether there is any time overlap in old data.

[0058] S4 loads the raw collected data into a sliding time window based on device identifiers and searches for old data with overlapping time. If duplicates are found, they are discarded; if they are valid corrected data, they are marked and retained. The cleaned data is then written in batches to the target database, and the scheduling weights and task fragments for the next round are adaptively adjusted.

[0059] In this embodiment, S4 relies on the semantic data cleaning unit and adopts a spatiotemporal normalization deduplication algorithm based on business semantic constraints to complete the accurate identification, cleaning and compliance correction of the original collected data. At the same time, it works in conjunction with the load awareness model to complete the iterative optimization of scheduling parameters, forming a complete closed loop of collection-cleaning-storage-scheduling optimization.

[0060] In this embodiment, the system continuously maintains a sliding time window in memory for segmentation management based on device ID, and the window length strictly follows the preset maximum physical latency threshold for each device. When new data is collected After entering the cleaning unit, it automatically retrieves all historical cached data within the current time window and filters out data whose timestamps meet the requirements. Overlapping old data in intervals For datasets with overlapping time periods, a unified calculation of the rate of change of values ​​is performed:

[0061] In this embodiment of the application, the rate of change is compared to the threshold. Instead of using fixed parameter design, adaptive learning optimization is performed based on the statistical distribution of long-term historical operating data from measurement points. The 99th quantile of the historical rate of change is typically selected as the dynamic threshold to accurately match the normal data fluctuation range under different power generation equipment and seasonal operating conditions. After calculating the rate of change, a hierarchical judgment logic is executed. When this is determined to be duplicate data entry in the system, the redundant data is discarded and a duplicate event log is recorded in the background; when When a data correction is deemed valid, such as manual correction or equipment malfunction reset, the new data content is fully preserved, a platform data correction alarm is triggered simultaneously, and a special audit mark is added to the data for easy traceability and verification later. For abnormal data that violates the physical operation laws of new energy sources, such as a reverse decrease in cumulative electricity consumption or exceeding limits, the system directly intercepts and filters the data and reports an alarm.

[0062] In this embodiment, for scenarios involving redundant data collection from multiple sources at the same measurement point, the system automatically performs cross-source data consistency cross-validation based on a preset data source trust priority configuration. It prioritizes the acceptance of valid data from high-priority data sources while simultaneously correcting erroneous data from low-priority data sources, further improving overall data consistency. After completing the entire cleaning and validation process, compliant and valid data is written in batches to the business target database. Simultaneously, key indicators such as the execution time, data write success rate, and channel concurrent load of this full-scale data collection task are statistically analyzed. The evaluation model of the load-aware execution cluster is updated concurrently, providing an iterative basis for the next round of data source response time, throughput score calculation, and task sharding adjustment.

[0063] like Figure 5 As shown in the figure, the entire process of sliding window caching, overlapping data retrieval, rate of change calculation, hierarchical determination, data processing and log recording is fully demonstrated, clearly presenting the core algorithm execution logic of the semantic cleaning unit.

[0064] In this embodiment, the original data semantic cleaning and correction are completed by using a spatiotemporal normalization deduplication algorithm and cross-source consistency verification. Combined with task operation indicators, the load awareness model is updated to achieve closed-loop optimization of data compliance entry and collection scheduling strategies.

[0065] Example 2 This invention relates to a dynamic routing and adaptive cleaning and acquisition system for multi-source heterogeneous data, such as... Figure 2 As shown, the core components include: a dual-modal dynamic configuration engine, an intelligent routing scheduling center, a semantic data cleaning unit, and a load-aware execution cluster.

[0066] Dual-modal dynamic configuration engine: Unlike traditional flat configuration, this solution designs a "real-time-historical" dual-modal configuration structure. In the configuration file (such as YAML), in addition to basic information, each measurement point must explicitly define its LifecycleMode. RealTime (real-time state): Binding logic: Automatically associate with high-concurrency memory queues and directly connect to the latest write interface of the time-series database.

[0067] Policy constraints: Force the disabling of any aggregation functions to ensure millisecond-level latency; enable "physical mutation detection".

[0068] Historical (historical state): Binding logic: Automatically associate with batch processing thread pools and dynamically construct historical query statements for big data platforms or time-series databases based on configuration.

[0069] Strategy constraints: AggregationStrategy (aggregation strategy, such as Max / Min / Avg), TimeWindow (time window), and TimeZoneAlign (time zone alignment rule) must be configured.

[0070] Intelligent Routing and Scheduling Center: This module parses the dual-modal configuration and executes dynamic routing.

[0071] Routing decision: Read the LifecycleMode of the measurement point. If it is RealTime, distribute it to the "Streaming Acquisition Channel"; if it is Historical, distribute it to the "Batch Acquisition Channel".

[0072] Protocol adaptation: For time-series database channels: Generate dedicated Line Protocol or SQL insert statements. For big data platform channels: Generate complex SQL / MapReduce tasks with time zone conversion and pre-aggregation functions.

[0073] Semantic data cleaning unit: Design a "spatiotemporal normalization deduplication algorithm based on business semantic constraints" tailored to the characteristics of new energy data.

[0074] Device-level time window locking: Define the "maximum physical latency threshold" for each type of device in the configuration.

[0075] Sliding window deduplication logic: The system maintains a sliding time window in memory, segmented by device ID, with a length of [missing information]. .

[0076] When new data Does the search window contain the information upon entry? Old data .

[0077] Slope check: If overlap exists, calculate the rate of change of the values. .

[0078] like (Conforming to physical laws, such as slow power change): This is considered a duplicate entry and should be discarded. And record a repeating event.

[0079] like (Abnormal mutation): If determined to be a valid correction (such as manually correcting an incorrect cumulative battery level), it will be retained. This triggers a "data correction alert," marking the data for auditing purposes.

[0080] Threshold adaptive learning: It is not a fixed value, but is dynamically adjusted according to the statistical distribution of the historical rate of change of the measuring point (such as taking the 99th percentile of the historical rate of change) to adapt to the normal fluctuation range under different equipment and different operating conditions.

[0081] Cross-source consistency verification: For test points with multiple sources, cross-validation is automatically performed based on the configured trusted source priority, prioritizing the acceptance of high-priority sources and repairing low-priority sources.

[0082] Load-aware execution cluster: Probe testing: During system startup and operation, lightweight queries are periodically sent to various data sources, and the real-time response time is recorded. and throughput To avoid instantaneous fluctuations, a sliding window is used to calculate the smoothed value.

[0083] Dynamic weight allocation: Calculate the scheduling weight for each data source based on probe test results. The specific calculation steps are as follows: Normalization: Convert indicators with different dimensions into dimensionless scores.

[0084] Response time score: ,in The score represents the minimum response time (i.e., the optimal value) among all current data sources; a higher score indicates a faster response.

[0085] Throughput Score: ,in This represents the maximum throughput among all current data sources; a higher score indicates stronger processing capabilities.

[0086] Weighted synthesis: Introduces configurable weight coefficients and ( Adjust the importance of response time and throughput according to business needs (e.g., increase when real-time requirements are high). Calculate the overall score: .

[0087] Normalized weights: Convert the overall score into scheduling weights that sum to 1. This weight reflects the relative processing capacity of each data source under the current load.

[0088] Adaptive sharding: Historical data collection tasks are no longer evenly sharded, but rather sharded according to weights. The number of data collection points allocated to each data source is dynamically adjusted. Let the total workload be N (the total number of device measurement points to be collected), then the data collection points allocated to each data source... The workload is: High-load nodes (low weight) automatically have their shards reduced, while low-load nodes (high weight) have their shards increased, thus achieving load balancing.

[0089] Back pressure feedback: When the response time of a certain data source... If the time exceeds a preset threshold (e.g., 5 seconds), the system automatically triggers a degradation strategy, such as pausing the retrieval of non-critical historical data from that source and only retaining real-time data collection to prevent a cascading failure. Once the data recovers, the weight gradually increases, and the workload automatically resumes.

[0090] Example 3 This invention provides a multi-source heterogeneous data dynamic routing and adaptive cleaning acquisition device 10, such as... Figure 6 As shown, the device includes: The mapping module 100 is used to load the measurement point configuration file, parse the life cycle mode attributes explicitly defined for each measurement point, dynamically map the measurement points to be collected to the streaming acquisition channel or batch acquisition channel according to the life cycle mode attributes, and load the physical constraint rules corresponding to the life cycle mode attributes. The testing module 200 is used to perform periodic probe tests on each data source, obtain real-time response time and throughput metrics, calculate the scheduling weight of each data source, and adjust the granularity of historical collection tasks and the allocation of concurrent resources. The acquisition module 300 is used to directly acquire real-time running data through the streaming acquisition channel and acquire historical cumulative data through the batch acquisition channel according to the preset aggregation strategy and time window to obtain the raw acquisition data. The retrieval module 400 is used to load the raw collected data into a sliding time window divided based on the device identifier, and to search for old data with overlapping time. If duplicates are found, they are discarded; if they are valid corrected data, they are marked and retained. The cleaned data is then written into the target database in batches, and the scheduling weights and task fragments are adaptively adjusted for the next round.

[0091] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0092] Example 4 To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0093] Example 5 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0095] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for dynamic routing and adaptive cleaning collection of multi-source heterogeneous data, characterized in that, include: S1, load the measurement point configuration file, parse the life cycle mode attributes explicitly defined for each measurement point, dynamically map the measurement points to be collected to the streaming acquisition channel or batch acquisition channel according to the life cycle mode attributes, and load the physical constraint rules corresponding to the life cycle mode attributes. S2 performs periodic probe tests on each data source to obtain real-time response time and throughput metrics, calculates the scheduling weight of each data source, and adjusts the granularity of historical data collection tasks and the allocation of concurrent resources. S3 directly acquires real-time running data through the streaming acquisition channel and acquires historical cumulative data through the batch acquisition channel according to the preset aggregation strategy and time window to obtain the raw acquisition data; S4 loads the raw collected data into a sliding time window based on device identifiers and searches for old data with overlapping time. If duplicates are found, they are discarded; if they are valid corrected data, they are marked and retained. The cleaned data is then written in batches to the target database, and the scheduling weights and task fragments for the next round are adaptively adjusted.

2. The method of claim 1, wherein, The loading of the measurement point configuration file parses the explicitly defined lifecycle mode attributes of each measurement point, dynamically maps the measurement points to be acquired to the streaming acquisition channel or batch acquisition channel according to the lifecycle mode attributes, and loads the physical constraint rules corresponding to the lifecycle mode attributes, including: Read the lifecycle mode attribute explicitly defined in the measurement point configuration file. When the lifecycle mode attribute is marked as real-time, map the measurement point to be collected to the streaming acquisition channel bound to the high-concurrency memory queue, and load the physical constraint rules that force the aggregation function to be disabled and enable physical mutation detection. When the lifecycle mode attribute is identified as historical, the measurement points to be collected are mapped to the batch collection channel bound to the batch processing thread pool, and physical constraint rules containing aggregation strategies, time windows and time zone alignment rules are loaded.

3. The method of claim 1, wherein, The process of performing periodic probe tests on each data source to obtain real-time response time and throughput metrics, calculating the scheduling weight of each data source, and adjusting the granularity of historical data collection tasks and the allocation of concurrent resources includes: Periodically send lightweight queries to various data sources as probe tests and record the real-time response time of each data source. and throughput A sliding window is used to calculate the smoothing value to eliminate instantaneous fluctuations; Based on the real-time response time and the throughput Calculate the normalized response time score and throughput score: in, The minimum response time among all current data sources. This represents the maximum throughput across all current data sources. Utilize configurable weighting coefficients and Calculate the overall score: The overall score is then normalized to a scheduling weight that sums to 1. ; According to the scheduling weight The number of collection points allocated to each data source is dynamically adjusted, assuming the total task is... Then it is assigned to the data source. The workload is: 。 4. The method according to claim 1, characterized in that, The process involves directly acquiring real-time operational data through a streaming acquisition channel and acquiring historical cumulative data through a batch acquisition channel based on a preset aggregation strategy and time window to obtain raw acquired data, including: If data acquisition is performed through the streaming acquisition channel, the latest data is directly retrieved from the latest write interface of the time series database, and format standardization processing is performed without applying any aggregation functions. If data collection is performed through the batch processing collection channel, a query statement containing time zone conversion logic and pre-aggregation functions is constructed according to the aggregation strategy and time window bound in the configuration file, and historical cumulative data after aggregation calculation is pulled from the big data platform or time series database as the raw collection data.

5. The method according to claim 1, characterized in that, The process involves loading the raw collected data into a sliding time window based on device identifiers, searching for overlapping old data, discarding duplicates, and marking and retaining valid corrected data. The cleaned data is then written in batches to the target database, and the scheduling weights and task fragments for the next round are adaptively adjusted. This includes: Maintain a sliding time window based on device ID slices in memory, with the window length being the maximum physical latency threshold defined in the configuration file. ; When new data Upon entry, does the search window contain a timestamp? old data If there is old data with overlapping time periods Then calculate the rate of change of the value: in, and These are the values ​​for the new data and the old data, respectively. and These are the timestamps for the new data and the old data, respectively. The rate of change of the numerical value With dynamically adjusted rate of change threshold If a comparison is made, If so, it is determined to be a duplicate entry and the new data is discarded. ,like The data is then deemed a valid correction and the new data is retained. Simultaneously trigger data correction alarms; if the data violates physical laws, it will be directly intercepted and an error will be reported. The cleaned data is written to the target database in batches, the time taken and success rate of this collection are recorded, and the load awareness model is updated for the next round of task sharding adjustment.

6. A dynamic routing and adaptive cleaning and acquisition device for multi-source heterogeneous data, characterized in that, include: The mapping module is used to load the measurement point configuration file, parse the life cycle mode attributes explicitly defined for each measurement point, dynamically map the measurement points to be collected to the streaming acquisition channel or batch acquisition channel according to the life cycle mode attributes, and load the physical constraint rules corresponding to the life cycle mode attributes. The testing module is used to perform periodic probe tests on each data source, obtain real-time response time and throughput metrics, calculate the scheduling weight of each data source, and adjust the granularity of historical data collection tasks and the allocation of concurrent resources. The acquisition module is used to directly acquire real-time running data through the streaming acquisition channel and acquire historical cumulative data through the batch acquisition channel according to the preset aggregation strategy and time window to obtain the raw acquisition data. The retrieval module loads the raw collected data into a sliding time window based on device identifiers, and searches for old data with overlapping time. If duplicates are found, they are discarded; if the data is valid and corrected, it is marked and retained. The cleaned data is then written in batches to the target database, and the scheduling weights and task fragments are adaptively adjusted for the next round.

7. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images