A method and system for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception

Abstract

The present application relates to the technical field of data fusion and real-time synchronization, and particularly relates to a multi-source heterogeneous data real-time alignment method and system based on dynamic timeliness perception. The method comprises the following steps: metadata exploration is performed on multi-source heterogeneous data sources, a dynamic timeliness evaluation model is constructed based on quantitative core timeliness indexes, and timeliness weights are given to each data; according to the timeliness weights, it is judged whether the data belongs to core data or non-core data, and difference processing is performed; according to the timeliness weights and preset business rules, the alignment priority of each data stream is dynamically adjusted and the computing resource is scheduled; when the aligned data is stored in the target data warehouse, the corresponding timeliness information is marked, and a unified data view is constructed. The present application accurately identifies the data value priority through dynamic timeliness perception, and realizes the optimal allocation of resources through the differential alignment strategy, thereby ensuring the traceability and visualization of the data and solving the problem of insufficient real-time performance of the traditional scheme.

Description

Technical Field

[0001] This invention relates to the field of data fusion and real-time synchronization technology, specifically to a method and system for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception. Background Technology

[0002] In applications such as building enterprise-level data infrastructure and real-time data lake warehouses, it is necessary to integrate and align heterogeneous data from multiple different data sources. These data sources vary significantly in update frequency, data transmission latency, and data freshness. Existing multi-source data alignment methods mainly target static, existing data, with a processing flow of one-time collection, cleaning, and alignment.

[0003] Patent application CN119046298A discloses a multi-source data synchronous update system based on a spatiotemporal digital base, including a data acquisition module, a spatiotemporal data management module, a data preprocessing module, a data synchronous update module, and an intelligent analysis module. Specifically, the data acquisition module acquires different types of multi-source data in real time; the spatiotemporal data management module performs spatiotemporal tagging and management on the acquired multi-source data; the data preprocessing module preprocesses the acquired multi-source data; the data synchronous update module uses intelligent algorithms to dynamically calculate the update frequency and temporal relationship of various data sources; and the intelligent analysis module performs intelligent analysis on the synchronously updated multi-source data, providing real-time feedback and early warning. Patent application CN120849427A discloses a method and system for dynamic synchronization of heterogeneous multi-source data based on a unified spatiotemporal framework, including the following steps: collecting heterogeneous multi-source data through a real-time monitoring interface and preprocessing it to eliminate differences in spatiotemporal reference and data format; using an incremental detection algorithm to identify changes in heterogeneous multi-source data, generating incremental update packets containing only the difference information of heterogeneous multi-source data and sending them to a message queue; the message queue performs hierarchical management of the incremental update packets and uses a distributed computing cluster to perform distributed parallel processing of the difference information; constructing a dynamic spatiotemporal index and locating conflicting data based on the spatiotemporal index; fusing the difference information and using a conflict resolution mechanism to fuse the conflicting data in the difference information, and finally storing it in a hierarchical storage architecture.

[0004] Existing real-time data alignment methods primarily acquire data from various data sources in batches within a preset collection period, then match, clean, and merge them according to unified field rules, ultimately generating a static alignment result set stored in a data warehouse. This approach cannot adapt to complex scenarios where data sources are continuously updated and the update frequency and timeliness of different data sources vary, leading to the problem of "outdated results immediately after completion," which in turn causes errors in subsequent decisions based on this data. Furthermore, as enterprises deepen their digital transformation, higher demands are placed on the real-time alignment capabilities of multi-source heterogeneous data, further highlighting the limitations of traditional technologies. Therefore, there is an urgent need to develop a real-time data alignment method that can solve the problems of inconsistent timeliness and alignment lag in multi-source data. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a real-time alignment method for multi-source heterogeneous data based on dynamic timeliness perception. Its core is to introduce a dynamic timeliness perception mechanism, combined with incremental alignment and real-time compensation strategies, to ultimately construct a unified data view with timeliness tags, thereby improving the real-time performance and effectiveness of multi-source heterogeneous data alignment, ensuring data consistency, and reducing the risk of decision-making errors caused by data delays or expiration.

[0006] Specifically, this is achieved through the following technical solution: In a first aspect, this application provides a method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness awareness, including the following steps: Metadata exploration is performed on multi-source heterogeneous data sources, and core timeliness indicators are quantified; a dynamic timeliness assessment model is constructed based on the core timeliness indicators; and timeliness weights are assigned to data sources and core fields based on the dynamic timeliness assessment model. Based on the timeliness weight of the data, it is determined whether the data belongs to core data or non-core data; for core data, real-time collection, cleaning, alignment and fusion are performed based on the latest data time point to ensure the timeliness of core data; for non-core data, asynchronous alignment is used for processing, and a real-time compensation mechanism is set to update the processed data to the target dataset and mark the compensation timestamp to identify that the data is the data written for later compensation. Based on the timeliness weight of the data and preset business rules, the alignment priority of each data stream is dynamically adjusted, and computing resources are scheduled according to the alignment priority to ensure that high-priority data streams with high timeliness requirements get computing resources first, and to avoid low-priority data streams blocking the alignment efficiency of core business data. When storing aligned data into the target data warehouse, mark the corresponding timeliness information and build a unified data view based on the marked data.

[0007] Furthermore, the metadata exploration of multi-source heterogeneous data sources includes identifying the field structure and data type of each data source; Furthermore, the core timeliness indicators include data freshness, update frequency, and transmission latency.

[0008] Furthermore, determining whether data belongs to core data or non-core data based on the timeliness weight of the data includes: A threshold T is dynamically set based on the real-time requirements of the business system, and the timeliness weight is compared with the threshold T. If the timeliness weight is ≥ T, the data is determined to be core data; if the timeliness weight is < T, the data is determined to be non-core data.

[0009] Furthermore, the preset business rule is to set a weight adjustment factor according to the importance level of the business domain to which the data belongs. Through the weight adjustment factor, the alignment priority of the data is forcibly increased or decreased. For example, for data streams involving high-value business domains such as bidding opportunities and sudden industry dynamics, the alignment priority is forcibly increased through the adjustment factor so that they are scheduled and processed first when computing resources are limited.

[0010] Furthermore, the computational resource scheduling based on alignment priority includes: Computational resources are prioritized for high-priority data streams, while low-priority data streams are allocated resources on demand. Specifically, this involves: establishing two independent processing thread pools for high and low priorities; high-priority data streams exclusively using high-performance thread resources to ensure real-time response; and low-priority data streams utilizing idle resources for asynchronous batch processing when system CPU utilization is below a preset threshold, achieving optimal resource allocation.

[0011] Furthermore, the timeliness information includes data collection time, alignment time, and data freshness level.

[0012] Furthermore, the unified data view is configured with query and filtering interfaces, which allow data users to set query or filtering conditions based on the timeliness information to obtain a subset of data that meets business timeliness requirements, ensuring that data users obtain accurate data that meets business timeliness requirements.

[0013] Secondly, this application provides a system for implementing the above-mentioned real-time alignment method for multi-source heterogeneous data, comprising: The timeliness assessment module is used to explore metadata from multi-source heterogeneous data sources, build a dynamic timeliness assessment model, and calculate and assign timeliness weights to each data point. The alignment strategy execution module is used to distinguish core data from non-core data according to the timeliness weight and perform differentiated processing. It performs real-time collection, clearing, alignment and fusion on core data, and performs asynchronous alignment and real-time compensation strategies on non-core data. The resource scheduling module is used to dynamically adjust the alignment priority of different data streams according to the timeliness weight, and to schedule computing resources according to the alignment priority. The view building module is used to create a unified data view with time-sensitive tags, and store the aligned data with time-sensitive information into the target data warehouse.

[0014] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-mentioned real-time alignment method for multi-source heterogeneous data based on dynamic timeliness perception.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness awareness.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention accurately identifies the value priority of data through dynamic timeliness perception; then, it achieves optimal resource allocation through differentiated alignment strategies, improving resource utilization and reducing system operating costs; finally, it outputs unified data with timeliness tags, ensuring data traceability and visualization, and solving the problem of insufficient real-time performance in traditional solutions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 Flowchart of a real-time alignment method for multi-source heterogeneous data based on dynamic timeliness perception; Figure 2 This is a schematic diagram of the dynamic timeliness assessment model structure; Figure 3 This is a unified data view display diagram with time-sensitive markers. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0020] The methods and systems provided in this application, in one embodiment, are as follows: Figure 1 As shown, firstly, heterogeneous metadata is explored between resource data (such as site physical information and supporting equipment parameters) and project implementation data (such as construction progress and on-site work order status) within the platform. A dynamic timeliness assessment model is used to quantify data freshness and update frequency, calculate timeliness weights, and assign them to corresponding data. A threshold T is set, and the timeliness weights are compared with the threshold T; if the timeliness weight ≥ T, the data is determined to be core data; if the timeliness weight < T, the data is determined to be non-core data.

[0021] Specifically, the dynamic timeliness assessment model is as follows: Figure 2 As shown, it includes a model input layer, a core indicator quantification layer, a weight fusion calculation layer, a model output layer, and an output docking layer. The model input layer includes metadata exploration result sets and real-time running status data of the data source. The core indicator quantification layer includes data freshness measurement units, update frequency quantification units, and transmission delay quantification units. The weight fusion extreme layer includes indicator normalization processing units, dynamic weighted fusion calculation units, and weight verification and dynamic calibration units. The model output layer includes data source-level timeliness weights and field-level timeliness weights.

[0022] Specifically, the core data includes (with extremely high real-time requirements): Project implementation data: including real-time progress feedback from the construction site, quality acceptance records of key processes, and sudden engineering alarm information. This data directly affects the timeliness of operation and maintenance scheduling decisions and must be collected and integrated in real time based on the latest point in time; High-frequency changing resource indicators: such as real-time power consumption data of base stations and transient alarms from environmental monitoring.

[0023] Non-core data includes (near real-time / asynchronous processing): resource data (physical attributes): including the geographical coordinates of the site, the outline of the main building, the material description of the tower, etc. Since the change frequency of this type of resource data is low (not occurring in real time), the system sets it to a low priority, processes it through asynchronous alignment, and marks it with a compensation timestamp to ensure eventual consistency; static archived data: such as scanned copies of completed project contracts, historical maintenance logs, etc.

[0024] Specifically, the implementation method for asynchronous alignment of non-core data includes the following steps: 1. Asynchronous task queue and buffer design Logical routing: When metadata probing reveals that data belongs to resource data with low timeliness weight, the system does not immediately trigger alignment calculation, but instead writes it to the "asynchronous pending queue".

[0025] Peak shaving and valley filling: When the system detects that the concurrency of project implementation data (core data) is too high, the asynchronous queue is automatically suspended and executed in sequence when computing resources are sufficient, so as to avoid blocking core marketing or scheduling business.

[0026] 2. Real-time compensation mechanism Difference capture: Set monitoring points for non-core data streams. If the data changes again during the asynchronous waiting period, the system will trigger the "overwrite compensation" logic to retain only the latest change instruction.

[0027] Compensation trigger: The system performs full verification every short period of time (e.g., once every 30 minutes) and actively captures residual drift data in the asynchronous queue for forced compensation and alignment to improve data consistency and accuracy.

[0028] 3. Asynchronous write strategy with tags Compensation timestamp: When data is finally written to the target data warehouse, the system must mark it with a "compensation timestamp" in addition to recording the original collection time. Timeliness level conversion: In the unified data view, this type of data will be assigned a specific "freshness level," indicating to the user that the resource data was generated asynchronously with compensation, rather than being delivered in absolute real-time.

[0029] 4. Resource scheduling and avoidance algorithm On-demand allocation: The system establishes an independent processing thread pool, and for asynchronous aligned tasks, computing resources are allocated only when the CPU utilization rate is below a preset threshold (such as 60%).

[0030] Priority dynamic escape: If a piece of resource data resides in the asynchronous queue for more than a preset time threshold (such as 2 hours), the system will automatically increase its priority, causing it to "escape" from the asynchronous stream and enter the real-time processing stream to prevent the data from expiring and becoming invalid.

[0031] Since project implementation data is directly linked to engineering delivery nodes, preset business rules increase its weight adjustment factor to forcibly elevate its data alignment priority. This assigns it a very high timeliness weight, prioritizing its processing when computing resources are limited, and allowing it to enter the core processing stream for millisecond-level real-time alignment. Resource data, due to its relatively stable physical properties, is monitored in real-time for its change frequency. Differentiated asynchronous alignment and periodic verification strategies are employed to significantly optimize the platform's computing resource allocation while ensuring the accuracy of operational visualization. Specifically: two independent processing thread pools are established for high and low priorities; high-priority alignment data streams exclusively utilize high-performance thread resources to ensure real-time response; low-priority alignment data streams utilize idle resources for asynchronous batch processing when the system CPU utilization is below a preset threshold, achieving optimal resource allocation.

[0032] When storing the aligned data into the target data warehouse, the corresponding timeliness information is marked. The timeliness information includes the data collection time, alignment time, and data freshness level. Based on the marked data, a unified data view is constructed. Data view architecture such as Figure 3 As shown, this view provides platform administrators with transparent data quality awareness capabilities: View Query Filtering Area (Area A): Supports operations and maintenance personnel to quickly filter out "high-freshness" project implementation data, or view the data alignment status of specific business domains (such as the two wings of business).

[0033] Basic Business Data Fields Area (Area B): Clearly displays core operation and maintenance fields such as the unique ID of the site address, the administrative region to which it belongs, and the operating data of supporting equipment.

[0034] Timeliness Marking Information Area (Area C): This is the core of the platform's monitoring. By displaying the original data collection timestamp and data freshness level, operations and maintenance personnel can immediately identify whether the current resource data is the latest "real-time disclosure" status or a near-real-time status based on periodic compensation, thereby effectively avoiding decision-making errors caused by data update lag.

[0035] In one embodiment, a system is provided for implementing a real-time alignment method for multi-source heterogeneous data, comprising: The timeliness assessment module is used to explore metadata from multi-source heterogeneous data sources, build a dynamic timeliness assessment model, and calculate and assign timeliness weights to each data point. The alignment strategy execution module is used to distinguish core data from non-core data according to the timeliness weight and perform differentiated processing. It performs real-time collection, clearing, alignment and fusion on core data, and performs asynchronous alignment and real-time compensation strategies on non-core data. The resource scheduling module is used to dynamically adjust the alignment priority of different data streams according to the timeliness weight, and to schedule computing resources according to the alignment priority. The view building module is used to create a unified data view with time-sensitive tags, and store the aligned data with time-sensitive information into the target data warehouse.

[0036] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described real-time alignment method for multi-source heterogeneous data.

[0037] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0038] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the above-described method for real-time alignment of multi-source heterogeneous data.

[0039] The invention and its embodiments have been described above illustratively. This description is not restrictive, and the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. The accompanying drawings are only one embodiment of the invention, and the actual structure is not limited thereto. No reference numerals in the claims should limit the scope of the claims. Therefore, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the invention, such design should fall within the scope of protection of this patent. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Multiple elements stated in the product claims may also be implemented by a single element through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception, characterized in that, Including the following steps: Metadata exploration is performed on multi-source heterogeneous data sources, and core timeliness indicators are quantified; a dynamic timeliness assessment model is constructed based on the core timeliness indicators; and timeliness weights are assigned to data based on the dynamic timeliness assessment model. Based on the timeliness weight of the data, determine whether the data belongs to core data or non-core data; For core data, real-time collection, cleaning, alignment, and fusion are performed based on the latest data point in time; For non-core data, an asynchronous alignment method is used for processing, and the processed data is updated to the target dataset and marked with a compensation timestamp; Based on the timeliness weight of the data and preset business rules, the alignment priority of each data stream is dynamically adjusted, and computing resources are scheduled according to the alignment priority. When storing aligned data into the target data warehouse, mark the corresponding timeliness information and build a unified data view based on the marked data.

2. The method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception according to claim 1, characterized in that, The core timeliness indicators include data freshness, update frequency, and transmission delay.

3. The method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception according to claim 1, characterized in that, The step of determining whether data belongs to core data or non-core data based on the timeliness weight of the data includes: A threshold T is dynamically set based on the real-time requirements of the business system, and the timeliness weight is compared with the threshold T. If the timeliness weight is ≥ T, the data is determined to be core data; if the timeliness weight is < T, the data is determined to be non-core data.

4. The method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception according to claim 1, characterized in that, The preset business rule is to set a weight adjustment factor based on the importance level of the business domain to which the data belongs, and to forcibly increase or decrease the alignment priority of the data through the weight adjustment factor.

5. The method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception according to claim 1, characterized in that, The process of scheduling computational resources based on alignment priority includes: Computing resources are allocated to data streams with high alignment priority first, and to data streams with low alignment priority on demand.

6. The method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception according to claim 1, characterized in that, The timeliness information includes data collection time, alignment time, and data freshness level.

7. A method for real-time alignment of multi-source heterogeneous data based on dynamic timeliness perception according to claim 6, characterized in that, The unified data view is configured with query and filtering interfaces, which allow data users to set query or filtering conditions based on the timeliness information to obtain a subset of data that meets business timeliness requirements.

8. A system for implementing the real-time alignment method for multi-source heterogeneous data as described in any one of claims 1-7, characterized in that, include: The timeliness assessment module is used to explore metadata from multi-source heterogeneous data sources, build a dynamic timeliness assessment model, and calculate and assign timeliness weights to each data point. The alignment strategy execution module is used to distinguish core data from non-core data according to the timeliness weight and perform differentiated processing. It performs real-time collection, clearing, alignment and fusion on core data, and performs asynchronous alignment and real-time compensation strategies on non-core data. The resource scheduling module is used to dynamically adjust the alignment priority of different data streams according to the timeliness weight, and to schedule computing resources according to the alignment priority. The view building module is used to create a unified data view with time-sensitive tags, and store the aligned data with time-sensitive information into the target data warehouse.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images