A multi-source heterogeneous data real-time fusion method, device and medium

Abstract

The application discloses a multi-source heterogeneous data real-time fusion method and device and medium, and belongs to the technical field of data fusion processing, and comprises the following steps: receiving real-time data streams from multiple heterogeneous data sources, using a semantic coding model to perform semantic alignment on fields of different data sources, and establishing a semantic mapping relationship of cross-source fields; inputting system running states into a deep reinforcement learning network to obtain a priority adjustment strategy of each data source; arbitrating conflict data according to the priority adjustment strategy to generate a unified data result after fusion; and updating parameters of the deep reinforcement learning network through a performance index of the unified data result. The output of the semantic coding model is not only used for establishing the semantic mapping relationship of the cross-source fields, but also input into the deep reinforcement learning network as an important component of the system running state, so that the priority adjustment strategy can make more accurate decisions based on the semantic understanding result.

Description

Technical Field

[0001] This invention relates to the field of data fusion processing technology, specifically to a method, device, and medium for real-time fusion of multi-source heterogeneous data. Background Technology

[0002] Real-time fusion of multi-source heterogeneous data faces increasingly severe challenges in scenarios such as smart cities, Industrial Internet of Things (IIoT), and real-time financial risk control. These scenarios are often deployed in distributed environments requiring high scalability, such as cloud computing platforms. With the exponential growth in the number of data sources and update frequency, traditional rule-based or static priority-based fusion methods are struggling to cope with dynamic conflicts in high-concurrency scenarios. Although mechanisms based on Change Data Capture (CDC), such as analyzing database logs, can achieve near real-time data synchronization, providing a foundation for applications such as real-time data warehouses, these mechanisms primarily focus on capturing and transmitting changed data and typically do not have built-in capabilities to resolve complex dynamic conflicts or semantic heterogeneity. Data integration, as a complex research field, has long faced challenges in handling heterogeneity, scale, and timeliness. As an important branch of the Internet of Things, the Industrial Internet of Things (IIoT) inherits its core characteristics and challenges, including a wide range of data sources, massive volume, high speed, and strong heterogeneity. In this complex environment, data inconsistency and conflicts become common problems, posing a severe test to the robustness, scalability, and accuracy of real-time fusion algorithms.

[0003] In recent years, Deep Reinforcement Learning (DRL) has demonstrated significant advantages in dynamic decision-making, such as its successful application in complex resource management. Unlike static rules, DRL can adaptively adjust its policies through online learning. However, existing DRL research largely focuses on single-objective optimization, and research on the common multi-objective trade-off between accuracy, latency, and resource consumption in multi-source heterogeneous scenarios remains insufficient. Furthermore, at the semantic level, the limitations of traditional alignment methods (such as word vectors) in handling polysemous words and cross-linguistic scenarios have been widely confirmed. While pre-trained language models (such as BERT) can improve semantic alignment accuracy, their large model size and computational requirements pose challenges to the deployment of high-concurrency, low-latency systems. Therefore, systematic optimization through model compression techniques such as knowledge distillation is necessary. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is: how to simultaneously solve the problems of dynamic data conflict resolution, cross-source semantic heterogeneity, and multi-objective performance trade-offs in high-concurrency multi-source heterogeneous data fusion scenarios, so as to achieve accurate, real-time, and efficient data fusion.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a method for real-time fusion of multi-source heterogeneous data, comprising the following steps:

[0007] It receives real-time data streams from multiple heterogeneous data sources, uses a semantic coding model to semantically align fields from different data sources, and establishes semantic mapping relationships between cross-source fields.

[0008] The semantic mapping relationship is used to detect data conflicts of the same entity, and the data records with conflicts are marked as conflict data. The conflict density is calculated and the time window size is adjusted according to the conflict density.

[0009] The system's operating status is input into the deep reinforcement learning network to obtain priority adjustment strategies for each data source;

[0010] Arbitrate conflicting data according to the priority adjustment strategy to generate a unified data result after fusion.

[0011] The parameters of the deep reinforcement learning network are updated by using performance metrics based on unified data results.

[0012] As a preferred embodiment of the real-time fusion method for multi-source heterogeneous data described in this invention, the semantic encoding model is a lightweight BERT model, and the specific steps include: compressing the BERT model into a multi-layer converter structure through knowledge distillation technology, and generating a semantic vector representation for each field.

[0013] As a preferred embodiment of the real-time fusion method for multi-source heterogeneous data described in this invention, the step of establishing a semantic mapping relationship between cross-source fields includes: calculating the semantic similarity of fields between different data sources; when the semantic similarity exceeds a judgment threshold, establishing a semantic mapping relationship between fields; the judgment threshold is adjusted according to the conflict density.

[0014] The steps for adjusting the judgment threshold include: statistically analyzing the conflict density within the current time window, calculating the historical conflict density mean and standard deviation; dividing the difference between the current conflict density and the historical conflict density mean by the standard deviation to obtain the standardized deviation value; performing a hyperbolic tangent function transformation on the standardized deviation value; and adding the benchmark threshold to the product of the hyperbolic tangent function transformation result and the adjustment amplitude to obtain the adjusted judgment threshold.

[0015] The beneficial effects of this preferred technical solution are: by combining statistical standardization with the hyperbolic tangent function, the judgment threshold can be smoothly adjusted, which improves the alignment accuracy when the conflict density is high and the recall rate when it is low.

[0016] As a preferred embodiment of the real-time fusion method for multi-source heterogeneous data described in this invention, the step of adjusting the time window size according to the conflict density includes: counting the number of conflicts detected within the current time window and comparing the number of conflicts with a preset threshold; when the number of conflicts exceeds the preset threshold, shortening the time window size to a first preset value; when the number of conflicts does not exceed the preset threshold, extending the time window size to a second preset value; and smoothing the number of conflicts by using a moving average during the adjustment of the time window size.

[0017] The beneficial effects of this preferred technical solution are: the sliding average smoothing process avoids unstable window adjustment and solves the system oscillation problem caused by instantaneous conflict spikes.

[0018] As a preferred embodiment of the real-time fusion method for multi-source heterogeneous data described in this invention, the system operating status includes data update frequency, historical conflict density, data quality score, and time window size; the deep reinforcement learning network is a deep Q-network, which adjusts the priority of each data source by increasing or decreasing the priority through an output priority adjustment strategy; the training of the deep reinforcement learning network adopts an experience replay mechanism and a target network update mechanism.

[0019] As a preferred embodiment of the real-time fusion method for multi-source heterogeneous data described in this invention, the step of arbitrating conflicting data according to the priority adjustment strategy includes: calculating the priority difference between the corresponding data sources of the conflicting data; when the priority difference is greater than the priority threshold, selecting the data from the higher priority data source as the fusion result; when the priority difference is less than or equal to the priority threshold, comparing the timestamps of the conflicting data and selecting the data with the latest timestamp as the fusion result; and introducing a timestamp verification mechanism to trigger a conflict warning when the timestamp difference exceeds a preset time threshold.

[0020] The beneficial effects of this preferred technical solution are: introducing timestamp arbitration when priorities are similar improves the accuracy of decision-making in boundary cases.

[0021] As a preferred embodiment of the real-time fusion method for multi-source heterogeneous data described in this invention, the performance indicators include conflict resolution success rate, fusion accuracy, processing latency, and resource consumption; the parameters of the deep reinforcement learning network are updated through a multi-objective reward function; the multi-objective reward function is the sum of the product of conflict resolution success rate and the first weight, the product of fusion accuracy and the second weight, minus the sum of the product of processing latency and the third weight, and the product of resource consumption and the fourth weight.

[0022] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multi-source heterogeneous data real-time fusion method.

[0023] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method for real-time fusion of multi-source heterogeneous data.

[0024] The beneficial effects of this invention are:

[0025] The deep integration of lightweight BERT semantic encoding, time window adjustment, and deep reinforcement learning networks produces a synergistic effect that goes beyond the simple superposition of individual technologies. The output of the semantic encoding model not only establishes semantic mapping relationships across source fields but also serves as a crucial component of the system's operational state, inputting into the deep reinforcement learning network. This enables the priority adjustment strategy to make more accurate decisions based on semantic understanding results. Simultaneously, conflict density statistics drive the dynamic adjustment of the time window size and influence the adaptive setting of the decision threshold through hyperbolic tangent function transformation, forming a linked optimization mechanism between conflict detection granularity and semantic alignment accuracy. This multi-layered state sharing and parameter linkage allows the system to achieve multi-objective collaborative optimization of accuracy, latency, and resource consumption in high-concurrency scenarios, solving the performance bottleneck problem caused by the single-objective optimization of traditional methods.

[0026] The dual arbitration mechanism of this invention automatically switches to timestamp comparison mode when the priority difference is less than or equal to the priority threshold. This design not only avoids decision-making difficulties when priorities are similar, but also provides additional conflict early warning capabilities through the timestamp verification mechanism. Combined with the smoothing of the number of conflicts by moving average, the system exhibits unexpected stability and robustness when facing instantaneous conflict spikes. Through the parameter update mechanism of the multi-objective reward function, the deep reinforcement learning network can continuously optimize the decision-making strategy when dealing with extreme conflict scenarios. Thus, in a complex environment where data dynamism, semantic heterogeneity, and multi-objective trade-offs coexist, high-quality real-time data fusion is achieved, providing reliable technical support for application scenarios such as smart cities, industrial IoT, and real-time financial risk control. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a real-time fusion method for multi-source heterogeneous data provided in one embodiment of the present invention.

[0029] Figure 2 This invention provides a BERT model structure for a real-time fusion method of multi-source heterogeneous data, as an embodiment of the present invention. Detailed Implementation

[0030] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0031] Example 1, referring to Figures 1-2 This is one embodiment of the present invention, which provides a method for real-time fusion of multi-source heterogeneous data, including the following steps S1 to S5:

[0032] S1. Receive real-time data streams from multiple heterogeneous data sources, use a semantic coding model to semantically align fields from different data sources, and establish a semantic mapping relationship between cross-source fields.

[0033] The semantic encoding model is a lightweight BERT model. The specific steps include: compressing the BERT model into a multi-layer transformer structure through knowledge distillation technology, and generating a semantic vector representation for each field.

[0034] In this embodiment, such as in an industrial IoT data fusion scenario, heterogeneous data is received from a relational database (MySQL database), a distributed document (MongoDB document), and a real-time sensor stream. MySQL contains English fields such as "order_id" and "amount", MongoDB contains Chinese fields such as "order number" and "amount", and the sensor stream contains a "cell" field (which may refer to either a battery cell or a network base station).

[0035] First, knowledge distillation is used to compress the large language model (BERT-base model (110M parameters)) into a 4-layer Transformer structure (28M parameters). Distillation loss function. Defined as:

[0036] ;

[0037] Where α is the balance coefficient, which in this embodiment is set to 0.7 to balance the cosine loss. and output difference loss The percentage of total distillation loss; Output vectors for the student model. This indicates that the vector is a A dimensional real vector; Output for the teacher model, This indicates that the vector is a A dimensional real vector; The learnable projection matrix has dimension 1. .

[0038] During the semantic vector generation process, field descriptions are input into the lightweight BERT model to generate 768-dimensional semantic vectors. The BERT model refers to... Figure 2 As shown, a BERT-BiLSTM-CRF hybrid architecture is adopted, which includes, from bottom to top, an input embedding layer, a Transformer coding layer, a BiLSTM layer, and a CRF output layer.

[0039] In the input embedding layer, The embedding vector representing the special classification label CLS is used to aggregate the semantic information of the entire input sequence; , , These represent word fragment embeddings from different fields of the first data source; , , These represent word fragment embeddings from different fields of the second data source; This represents the embedding vector of the Special Separator (SEP), used to separate fields from different data sources.

[0040] The Transformer encoding layer is compressed from 12 layers in the original BERT-base model to 4 layers through knowledge distillation. The dense cross connections in the figure represent the information interaction between different positions in the multi-head self-attention mechanism. , , , , , , , These represent different 768-dimensional hidden state vectors after Transformer encoding.

[0041] The BiLSTM layer employs a bidirectional long short-term memory network structure to further enhance the model's ability to model the context of field semantics. The CRF layer, as the output layer of the conditional random field, models the transition constraints between labels and outputs the final sequence labeling results. Indicates the start of the field boundary marker. This indicates a marker within a field boundary, with 'O' indicating a marker for a non-critical field region. For example, generating the first semantic vector for "order_id". Generate a second semantic vector for the "order number". . It is a 768-dimensional real vector space. For the polysemous word "cell", combining its data type and contextual information, a battery cell is generated in the battery monitoring scenario. In network monitoring scenarios, network base stations are generated ( This enables context-sensitive semantic mapping.

[0042] Table 1: Performance Comparison of Lightweight BERT

[0043] Model Parameters Alignment precision Inference latency BERT-base 110M 94.9% 23.4ms (After distillation loss) BERT-distilled 28M 93.7% 11.2ms

[0044] Experiments show that the compressed model achieves 93.7% accuracy in cross-language alignment tasks, a decrease of only 1.2pp compared to the original BERT, while improving inference speed by 2.1 times.

[0045] The steps to establish a semantic mapping relationship between cross-source fields include A1 to A3:

[0046] A1. Calculate the semantic similarity of fields between different data sources;

[0047] In this embodiment, semantic similarity sim() is calculated, taking "order_id" and "customer number" as an example:

[0048] ;

[0049] Similarly, in another alternative implementation, the similarity between "order_amount" and "order amount" is calculated to be 0.895, and the similarity between "cell_voltage" and "battery voltage" is calculated to be 0.887.

[0050] A2. When the semantic similarity exceeds the judgment threshold, a semantic mapping relationship between fields is established;

[0051] The current judgment threshold is set to τ(t) = 0.82 based on expert experience. Since sim(order_id, customer number) = 0.912 > 0.82, a semantic mapping relationship M(order_id, customer number) = 1 is established. Similarly, since the similarity between "order_amount" and "order amount" is 0.895 > 0.82, a semantic mapping relationship M("order_amount", "order amount") = 1 is established. Furthermore, the similarity between "cell_voltage" and "battery voltage" is 0.887 > 0.82, so a semantic mapping relationship is also established.

[0052] A3. The determination threshold is adjusted according to the conflict density;

[0053] The steps for adjusting the threshold include A3.1 to A3.4:

[0054] A3.1 Calculate the conflict density within the current time window, and then calculate the historical conflict density mean and standard deviation.

[0055] Within the current time window Δt = 1.0 seconds, the system detected 15 field conflict events. The conflict density was calculated. :

[0056] ;

[0057] Calculate the average historical conflict density based on historical data from the past 100 time windows. Conflict / second, standard deviation Conflicts per second.

[0058] A3.2 Divide the difference between the current conflict density and the historical conflict density mean by the standard deviation to obtain the standardized deviation value;

[0059] Divide the difference between the current conflict density and the historical mean by the standard deviation:

[0060] Standardized deviation value: .

[0061] A3.3. Perform hyperbolic tangent function transformation on the standardized deviation values;

[0062] Perform hyperbolic tangent transformation on the standardized deviation values:

[0063] .

[0064] A3.4. Add the product of the baseline threshold, the hyperbolic tangent function transformation result, and the adjustment amplitude to obtain the adjusted judgment threshold. ;

[0065] Baseline threshold The result of the hyperbolic tangent function transformation and the adjustment amplitude Adding products that are 0.1:

[0066] .

[0067] When setting the baseline threshold, based on general experience in semantic similarity matching, 0.8 can achieve a good balance between precision and recall, ensuring that most semantically equivalent fields are correctly identified while effectively filtering field pairs with large semantic differences.

[0068] Because the current conflict density is higher than the historical average, the system automatically raises the judgment threshold from 0.82 to 0.875, increasing the accuracy requirements for semantic alignment. In the next round of field alignment, only field pairs with a semantic similarity exceeding 0.875 will establish a semantic mapping relationship, thereby ensuring stricter semantic matching standards in a high-conflict environment and reducing the degradation of data fusion quality caused by semantic alignment errors.

[0069] S2. Detect data conflicts of the same entity through the semantic mapping relationship, mark conflicting data records as conflicting data, calculate the conflict density and adjust the time window size according to the conflict density;

[0070] Within the current time window Δt = 1.0 seconds, a request for the entity is received. Concurrent updates:

[0071] Source 1 (MySQL):

[0072] ;

[0073] Source 2 (MongoDB):

[0074] ;

[0075] By using the semantic mapping relationships M(customer_id, customer ID) = 1 and M(order_amount, order amount) = 1, it can be identified that these two records refer to the same entity. The two records have the same semantic field, but their values ​​differ (1500 vs 1520). Mark this situation as a data conflict and label both records as conflicting data.

[0076] Similarly, within the same time window, the system also detected entities. ,entity ,entity A total of 18 entities have field conflicts, with a total number of conflicts C=18.

[0077] The steps for adjusting the time window size based on conflict density include B1 to B4:

[0078] B1. Count the number of conflicts detected within the current time window and compare the number of conflicts with a preset threshold;

[0079] Based on human experience, a preset threshold T=15 is established. Then, the number of collisions detected within the current time window Δt=1.0 seconds is counted as C=18. The number of collisions is compared with the preset threshold: C=18>T=15, indicating that the current number of collisions exceeds the preset threshold.

[0080] B2. When the number of conflicts exceeds a preset threshold, shorten the time window size to a first preset value;

[0081] Since the number of conflicts C > T, shorten the time window size from the current Δt = 1.0 second to the first preset value of 0.5 seconds. The basis for setting the first preset value of 0.5 seconds is to improve the time granularity of conflict detection by shortening the time window, reduce the conflict accumulation within a single window, and thus enhance the timeliness of conflict handling.

[0082] Within the next time window Δt = 0.5 seconds, re - count the number of conflicts. Assume that the detected number of conflicts C = 8. Since C = 8 < T = 15, it meets the condition of step B3.

[0083] B3. When the number of conflicts does not exceed the preset threshold, extend the time window size to a second preset value;

[0084] When the number of conflicts C = 8 does not exceed the preset threshold T = 15, extend the time window size from the current 0.5 seconds to the second preset value of 1.0 seconds. The basis for setting the second preset value to 1.0 seconds is to ensure that a larger time window is adopted when the conflict density is low to improve the processing efficiency.

[0085] B4. When adjusting the time window size, smooth the number of conflicts through a moving average;

[0086] To avoid unstable adjustment of the window size caused by instantaneous conflict spikes, smooth the number of conflicts through a moving average. Set the moving window length to 5 based on expert experience, and average the number of conflicts in the recent 5 time windows:

[0087] Assume that the number of conflicts in the recent 5 windows is [18, 12, 16, 20, 14] respectively, and calculate the moving average: <00​​​​​​​​​​​​​​​​​​​​​​​The adjusted time window size of 0.5 seconds will be used as part of the system's operating status and passed to subsequent deep reinforcement learning networks for priority decision-making.

[0093] S3. Input the system running status into the deep reinforcement learning network to obtain the priority adjustment strategy for each data source.

[0094] The system operating status includes data update frequency, historical conflict density, data quality score, and time window size;

[0095] It's important to know that the data update frequency (fupdate) is the average update frequency of each data source over the past minute.

[0096] MySQL database: 120 records / minute → normalized to fupdate,1=0.6;

[0097] MongoDB document: 80 records / minute → normalized to fupdate,2=0.4;

[0098] Sensor stream: 200 streams / minute → Normalized to fupdate,3=1.0.

[0099] The historical conflict density (fconflict) is the moving average conflict density calculated based on step B4:

[0100] Conflicts per second, normalized to fconflict=0.8.

[0101] The data quality score (D) is a comprehensive assessment based on data integrity, consistency, and timeliness.

[0102] MySQL: Integrity 95%, Consistency 92%, Timeliness 98% → D1=0.95;

[0103] MongoDB: Integrity 88%, Consistency 89%, Timeliness 94% → D2=0.90;

[0104] Sensor flow: Integrity 99%, Consistency 85%, Timeliness 99% → D3=0.94.

[0105] The time window size (twindow) is the currently adjusted time window size.

[0106] twindow = 0.5 seconds, normalized to 0.25.

[0107] Constructing the system state vector: , It is a 4-dimensional real vector space.

[0108] The deep reinforcement learning network is a deep Q-network, which adjusts the priority of each data source by increasing or decreasing the priority through an output priority adjustment strategy;

[0109] It is important to know that the deep Q-network structure consists of: 4 neurons receiving state vectors as the input layer; 64 neurons with ReLU activation function as hidden layer 1; 32 neurons with ReLU activation function as hidden layer 2; and 15 neurons (3 data sources × 5 actions) as the output layer.

[0110] Output the Q-value matrix corresponding to each data source :

[0111]

[0112] To execute an action, an ε-greedy strategy is used to select the action. For calculating the Q-value for a MySQL database, For calculating the Q-value for MongoDB documents, This section calculates the Q-value for the sensor stream. The current greedy strategy selects actions of 0.1: MySQL data source selection action +1 (priority increased by 1 level); MongoDB data source selection action -1 (priority decreased by 1 level); Sensor stream selection action 0 (priority remains unchanged). Based on the decision results, the priority adjustment strategy is output. This is the system state vector.

[0113] The training of the deep reinforcement learning network employs an experience replay mechanism and a target network update mechanism.

[0114] In this embodiment, the system maintains an experience replay buffer B with a capacity of 10,000. After priority adjustment, the system calculates the multi-objective reward function:

[0115] Then, weights are set based on human experience. Through a multi-objective reward function calculate:

[0116] ;

[0117] in, This represents the ratio of the number of successfully resolved conflicts to the total number of conflicts. This indicates the proportion of the fusion result that is consistent with the baseline data. This represents the average response time from collision detection to fusion completion. This indicates the combined utilization of CPU and memory.

[0118] experience tuples Stored in experience playback buffer B:

[0119] ; (Actions corresponding to three data sources); ; (Next state).

[0120] in, Let be the system state vector. Adjust the action based on the priority selected at the current moment. Let be the multi-objective reward value at time t. Let be the system state vector at time t+1.

[0121] In the target network update mechanism, the system maintains the main network parameters. and target network parameters After processing every 32 batches, training is performed by randomly sampling empirical samples of batch size 64 from the empirical replay buffer B.

[0122] Calculate the current state Q value using the main network: ;

[0123] Target Q-value calculation: Calculated using the target network:

[0124] ;

[0125] Among them, discount factor , Let Q be the target Q value at time t, and use it as the training label. In the state The set of all possible actions. To select the action that maximizes the Q value;

[0126] In discount factor When setting the discount factor, it represents the standard hyperparameter, with a value range of [0,1). A value of 0.99 is a typical setting, indicating that the system highly values ​​long-term benefits. In real-time data fusion scenarios, long-term performance stability needs to be considered, hence the choice of a discount factor of 0.99.

[0127] Calculate the mean square error loss:

[0128] ;

[0129] In the formula, L is the loss function value, used to measure the prediction error, and N is the batch size. In this embodiment, a total of 64 experience samples are used for training, so it is set to 64. Let Q be the target Q value for the i-th training sample. The main network parameters are used to predict the Q-value of the i-th sample. Let represent the state and action of the i-th training sample, respectively.

[0130] Update the main network parameters θ using the Adam optimizer, and copy the main network parameters to the target network every 1000 training steps: .

[0131] Through this mechanism, deep networks (DQN networks) can learn effective priority adjustment strategies in high-conflict-density environments, achieving dynamic decision optimization under multi-objective optimization goals.

[0132] After processing each batch of data or after a fixed period of time, sample a small batch to update the DQN weights and periodically synchronize them to the target network. If the conflict rate is abnormally high (>90%), the initial reward fluctuations will be significantly increased, but it can often converge to a stable policy after tens of thousands of training steps.

[0133] S4. Arbitrate conflicting data according to the priority adjustment strategy to generate a unified data result after fusion;

[0134] The steps for arbitrating conflicting data according to the priority adjustment strategy include C1 to C4:

[0135] C1. Calculate the priority difference between the data sources corresponding to conflicting data;

[0136] For entities Order amount conflict:

[0137] Conflicting data 1: Source: MySQL, "order_amount": 1500, ;

[0138] Conflicting data 2: Source: MongoDB, "Order Amount": 1520 .

[0139] It's important to know that MySQL, as a relational database, possesses ACID transaction properties (atomicity, consistency, isolation, and durability), with strict data integrity constraints and high data consistency and reliability. Therefore, in this embodiment, it is given a higher priority of 4. MongoDB, as a document-oriented NoSQL database, has relatively relaxed data constraints, and data version conflicts may occur in high-concurrency write scenarios. Therefore, in this embodiment, it is given a lower priority of 1. It's also important to know that "order_amount" is directly retrieved from MySQL as 1500, and "order amount" is directly retrieved from MongoDB as 1520.

[0140] Calculate the priority difference:

[0141] ;

[0142] in: This is the priority difference, used to determine the arbitration strategy. For the priority of data source i, This represents the priority of data source j.

[0143] C2. When the priority difference is greater than the priority threshold, the data from the higher priority data source is selected as the fusion result;

[0144] Priority thresholds are set based on expert experience. .because The condition that the priority difference is greater than the priority threshold is met.

[0145] According to the decision-making rules:

[0146] ;

[0147] in: The data values ​​of the final fusion result. These are the conflicting data values ​​from data sources i and j, respectively. As a priority threshold, For data record k, use the timestamp.

[0148] because Select a high-priority data source The data is used as the fusion result:

[0149] ;

[0150] Priority for the data source MySQL Priority for the data source MongoDB.

[0151] C3. When the priority difference is less than or equal to the priority threshold, compare the timestamps of the conflicting data and select the data with the latest timestamp as the fusion result.

[0152] For entities Another set of conflicting data:

[0153] Conflicting data 1: Source: MySQL, "cell_voltage": 3.75V. , ;

[0154] Conflicting data 2: Source: Sensor stream, "cell": 3.72V. , .

[0155] In this embodiment, the conflict data 1 Still set to 4, while in conflict data 2 It is more lenient than MySQL, so it is set to 3.

[0156] Calculate priority difference ,because This triggers the timestamp arbitration mechanism. The first timestamp is compared. Second timestamp :

[0157] (Unix timestamp);

[0158] (Unix timestamp).

[0159] because Select the sensor stream data with the updated timestamp:

[0160] V; where, For data The corresponding timestamp.

[0161] C4. Introduce a timestamp verification mechanism to trigger a conflict warning when the timestamp difference exceeds a preset time threshold.

[0162] Introduce a timestamp verification mechanism:

[0163] ;

[0164] in: The timestamp difference is used to detect the timeliness of data. The current time window size, obtained from step B2, is 0.5 seconds. The preset time threshold is the minimum of twice the window size and 1 second.

[0165] Calculate timestamp differences:

[0166] ;

[0167] Preset time threshold:

[0168] ms;

[0169] because The timestamp verification passed, and the data timeliness meets the requirements.

[0170] If the timestamp difference exceeds a preset time threshold, the system will trigger a conflict warning mechanism, record the abnormal event, and notify the operations and maintenance personnel.

[0171] S5. Update the parameters of the deep reinforcement learning network by using performance metrics from unified data results;

[0172] Performance metrics include conflict resolution success rate, fusion accuracy, processing latency, and resource consumption.

[0173] The parameters of the deep reinforcement learning network are updated using a multi-objective reward function.

[0174] The multi-objective reward function is the sum of the product of conflict resolution success rate and the first weight, the product of fusion accuracy and the second weight, minus the sum of the product of processing latency and the third weight, and the product of resource consumption and the fourth weight.

[0175] First, four key performance indicators (KPIs) of the unified data results are calculated. The conflict resolution success rate is measured by the ratio of the number of successfully resolved conflicts to the total number of detected conflicts within the current time window. The formula for calculating the conflict resolution success rate is:

[0176] ;

[0177] SR represents the conflict resolution success rate. The number of conflicts successfully resolved. This represents the total number of collisions detected. Within the current time window, this represents the total number of collisions detected. The number of conflicts successfully resolved (An alert was triggered due to an abnormal timestamp in one conflict), and the conflict resolution success rate was calculated. The success rate of conflict resolution after standardization .

[0178] Fusion accuracy is evaluated by comparing the system's fusion results with a benchmark dataset. The formula for calculating fusion accuracy is:

[0179] ;

[0180] Where FA represents the fusion accuracy. To ensure the number of fusion results consistent with the baseline data, This represents the total amount of fused data. Based on validation using a benchmark dataset with expert annotations, the total amount of fused data... Correct fusion result (where the entity) (The fusion results have biases), and the fusion accuracy is calculated. Fusion accuracy after standardization .

[0181] Processing latency reflects the system's response speed from collision detection to fusion completion. The formula for calculating average processing latency is:

[0182] ;

[0183] Where PL represents the average processing latency, and N is the number of conflicts to be processed. The timestamp indicating the completion of merging the i-th conflict. Let be the timestamp of the start of detection for the i-th conflict. Counting 17 successfully processed conflicts, the total processing time was 850ms, and the average processing latency was... ms, the average processing latency after standardization based on 100ms. .

[0184] Resource consumption takes into account both CPU and memory usage. The formula for calculating comprehensive resource consumption is:

[0185] ;

[0186] Wherein, RC represents the overall resource consumption rate. Set the CPU weight to 0.6. The memory weight is set to 0.4, and the two weights involved in the comprehensive resource consumption calculation formula are set based on expert experience. Current CPU usage This is the system's maximum CPU capacity. This represents the current memory usage. This represents the system's maximum memory capacity. Monitoring data shows that CPU utilization is 75% and memory utilization is 68%. The calculated overall resource consumption rate is... .

[0187] Based on the above four performance metrics, a multi-objective reward function is constructed for updating DQN network parameters. The multi-objective reward function is defined as follows:

[0188] ;

[0189] in, This represents the multi-objective reward value at time t. In the multi-objective reward function formula, the four sets of weights are set by expert experience, as exemplified below. The first weight for conflict resolution success rate is set to 0.3. The second weight for fusion accuracy was set to 0.3. The third weight for handling latency is set to 0.2. The fourth weight for resource consumption is set to 0.2. Substituting this into the calculation, we get Rt = 0.321.

[0190] Deep reinforcement learning network parameter updates use the gradient descent method, which calculates... with empirical tuples Stored in the experience replay buffer. The parameter update formula is:

[0191] ;

[0192] in, This represents the main network parameters, including the weights and bias matrices. The learning rate is set to 0.001 to control the step size of parameter updates. This represents the gradient of the loss function with respect to the main network parameters. Loss function Defined as:

[0193] ;

[0194] in, This indicates that the experience replay buffer stores historical experience data. The target network parameters are used to calculate the target Q-value. This is a discount factor used to balance immediate rewards and future rewards. This represents the expected calculation of samples in the experience replay buffer.

[0195] In subsequent decisions, the updated DQN network achieved an average reward value increase from 0.287 to 0.321, maintained a conflict resolution success rate above 94%, a fusion accuracy stable within the 93-95% range, and a processing latency controlled within 50ms. Through continuous performance feedback and parameter update mechanisms, the system achieved dynamic adaptive adjustment under multi-objective optimization constraints, effectively improving the overall performance and robustness of heterogeneous data fusion.

[0196] Example 2 is an embodiment of the present invention. This embodiment provides an electronic device applicable to a real-time fusion method for multi-source heterogeneous data, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize a real-time fusion method for multi-source heterogeneous data as proposed in the above embodiment.

[0197] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements a real-time fusion method for multi-source heterogeneous data as proposed in the above embodiments.

[0198] The storage medium proposed in this embodiment and the method for real-time fusion of multi-source heterogeneous data proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0199] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0200] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for real-time fusion of multi-source heterogeneous data, characterized in that, Includes the following steps: It receives real-time data streams from multiple heterogeneous data sources, uses a semantic coding model to semantically align fields from different data sources, and establishes semantic mapping relationships between cross-source fields. The semantic mapping relationship is used to detect data conflicts of the same entity, and the data records with conflicts are marked as conflict data. The conflict density is calculated and the time window size is adjusted according to the conflict density. The system's operating status is input into the deep reinforcement learning network to obtain priority adjustment strategies for each data source; Arbitrate conflicting data according to the priority adjustment strategy to generate a unified data result after fusion. The parameters of the deep reinforcement learning network are updated using performance metrics based on unified data results; The steps to establish a semantic mapping relationship between cross-source fields include: Calculate the semantic similarity of fields between different data sources; The formula for calculating semantic similarity sim() is expressed as: ； in, The first semantic vector, This is the second semantic vector; When the semantic similarity exceeds the judgment threshold, a semantic mapping relationship is established between the fields; The determination threshold is adjusted according to the conflict density; The adjustment steps for the determination threshold include: Calculate the conflict density within the current time window, and then calculate the historical conflict density mean and standard deviation. The standard deviation is obtained by dividing the difference between the current conflict density and the historical conflict density mean by the standard deviation. The standardized deviation value is expressed as: ; in, Given the current conflict density, This represents the average density of historical conflicts. Standard deviation; The standardized deviation value is transformed by the hyperbolic tangent function, and the formula is as follows: ； The adjusted judgment threshold is obtained by adding the product of the baseline threshold, the hyperbolic tangent function transformation result, and the adjustment range. Adjusted judgment threshold The calculation formula is expressed as: ； in, As the baseline threshold, To adjust the amplitude; The steps for adjusting the time window size based on conflict density include: The number of conflicts detected within the current time window is counted, and the number of conflicts is compared with a preset threshold. When the number of conflicts exceeds a preset threshold, the time window size is shortened to a first preset value; When the number of conflicts does not exceed a preset threshold, the time window size is extended to a second preset value; When adjusting the size of the time window, the number of conflicts is smoothed by using a moving average. The steps for arbitrating conflicting data according to the priority adjustment strategy include: Calculate the priority difference between the data sources corresponding to conflicting data; When the priority difference is greater than the priority threshold, the data from the higher priority data source is selected as the fusion result; When the priority difference is less than or equal to the priority threshold, the timestamps of the conflicting data are compared, and the data with the latest timestamp is selected as the fusion result. A timestamp verification mechanism is introduced, which triggers a conflict warning when the timestamp difference exceeds a preset time threshold.

2. The method for real-time fusion of multi-source heterogeneous data as described in claim 1, characterized in that, The semantic encoding model is a lightweight BERT model, and the specific steps include: The BERT model is compressed into a multi-layer transformer structure using knowledge distillation technology, and a semantic vector representation is generated for each field.

3. The method for real-time fusion of multi-source heterogeneous data as described in claim 2, characterized in that, The system operating status includes data update frequency, historical conflict density, data quality score, and time window size; The deep reinforcement learning network is a deep Q-network, which adjusts the priority of each data source by increasing or decreasing the priority through an output priority adjustment strategy; The training of the deep reinforcement learning network employs an experience replay mechanism and a target network update mechanism.

4. The method for real-time fusion of multi-source heterogeneous data as described in claim 3, characterized in that, The performance metrics include conflict resolution success rate, fusion accuracy, processing latency, and resource consumption. The parameters of the deep reinforcement learning network are updated using a multi-objective reward function. The multi-objective reward function is the sum of the product of conflict resolution success rate and the first weight, the product of fusion accuracy and the second weight, minus the sum of the product of processing latency and the third weight, and the product of resource consumption and the fourth weight.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the real-time fusion method for multi-source heterogeneous data as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the real-time fusion method for multi-source heterogeneous data as described in any one of claims 1 to 4.

Images