Multi-source monitoring data information fusion processing method and system

Through adaptive cleaning and iterative optimization algorithm fusion processing, combined with dynamic model and Kalman gain, data quality and flexibility problems in multi-source data processing are solved, and more accurate and stable data fusion results are achieved.

CN118981744AInactive Publication Date: 2024-11-19BEIJING KEJIA LONGBANG TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411070328.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

There are uneven data quality and lack of flexibility and adaptability in multi-source data processing, which affects the accuracy and reliability of fusion results.

Method used

The multi-source monitoring data information fusion processing method is adopted, and the monitoring data is obtained from different data sources, and the iterative cleaning and iterative optimization algorithm fusion processing is carried out, and the data fusion is combined with dynamic models and Kalman gain is adjusted in real time.

Benefits of technology

It significantly improves the comprehensiveness and accuracy of data processing, effectively removes noise and outliers, improves the intelligence level and efficiency of data processing, ensures the accuracy and stability of the fusion results, and adapts to changes in data sources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981744A_ABST
    Figure CN118981744A_ABST
Patent Text Reader

Abstract

The invention provides a multi-source monitoring data information fusion processing method and system. The method comprises the following steps: acquiring monitoring data of a target from different data sources; self-adaptive cleaning is carried out on the monitoring data; and performing fusion processing on the monitoring data of different data sources through an iterative optimization algorithm to obtain a fusion result. According to the method, the monitoring data are flexibly and automatically cleaned through the adaptive cleaning technology, noise and abnormal values can be effectively removed, the data quality is ensured, meanwhile, the intelligent level and efficiency of data processing are improved, the multi-source monitoring data are fused through the iterative optimization algorithm, and the data processing efficiency is improved. According to the method, the data fusion process can be continuously adjusted and optimized, so that the accuracy and the stability of the fusion result are improved, the iteration process is favorable for adapting to differences and changes among different data sources, continuous optimization of the fusion result is ensured, and the more accurate and reliable fusion result can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for fusion processing of multi-source monitoring data information. Background Art

[0002] In the context of big data, the speed at which data is generated is growing explosively, and the number of data sources in different fields and industries is also increasing. As a result, the problem of processing multi-source data has also emerged. For example, in many fields such as the Internet, social security, and medical health, there are problems with processing multi-source data.

[0003] At present, there are still many challenges in multi-source data processing. On the one hand, the quality of multi-source data is uneven, and there are omissions and errors. If it is not fully cleaned and processed, it will directly affect the fusion results. On the other hand, since the data source may fluctuate or be abnormal due to environmental changes, equipment failures, data transmission problems or other external factors, and existing technologies may not be able to dynamically adjust the fusion strategy according to the real-time changes of the data source, this means that in the data fusion process, they often lack flexibility and adaptability. If the data fusion system cannot detect these changes in real time and adjust its fusion strategy accordingly, the accuracy and reliability of the fusion results will be affected. Summary of the invention

[0004] Based on this, the purpose of the present invention is to propose a multi-source monitoring data information fusion processing method and system to solve the above-mentioned problems.

[0005] According to the multi-source monitoring data information fusion processing method proposed by the present invention, the method comprises:

[0006] Obtain monitoring data of the target from different data sources;

[0007] Adaptively cleaning the monitoring data;

[0008] The monitoring data from different data sources are fused through an iterative optimization algorithm to obtain a fusion result.

[0009] Furthermore, the step of fusing the monitoring data from different data sources through an iterative optimization algorithm to obtain a fusion result includes:

[0010] Set the initial state and initial error covariance matrix;

[0011] Performing time synchronization and spatial alignment on the monitoring data from different data sources;

[0012] By using an iterative optimization algorithm, combined with the dynamic model and the previous state, the multi-source monitoring data at each time point is fused, and the fusion result at each time point is obtained.

[0013] Furthermore, the step of using an iterative optimization algorithm, combining a dynamic model and a previous state, fusing the multi-source monitoring data at each time point, and obtaining a fusion result at each time point includes:

[0014] For each time point, calculate the previous state estimate x i-1 The difference between the monitoring data of each data source and the residual of each data source is obtained, where the previous state estimate x in the first iteration is i-1 is the initial state x0;

[0015] Calculating a Kalman gain for each data source, wherein the Kalman gain reflects the weight of each data source when updating the state estimate;

[0016] The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i ;

[0017] According to the preset convergence threshold and the maximum number of iterations N, the current state of each data source is iteratively updated to obtain the final current state estimation value of each data source;

[0018] The final current state estimation values ​​of all data sources are weighted averaged to obtain the fusion result X of the monitoring data of all data sources.

[0019] Furthermore, the current state estimation value x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i The steps include:

[0020] The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i , the calculation formula is: i =x i-1 +K i ·(zH i ·x i-1 ), where z is the monitoring data value of the corresponding data source.

[0021] Furthermore, the current state estimation value x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i The steps include:

[0022] Calculate and update the current error covariance matrix P i , the calculation formula is: i =(IK i ·H i )·P i-1 , where I is the identity matrix, Pi-1 is the previous error covariance matrix.

[0023] Furthermore, the step of calculating the Kalman gain of each data source includes:

[0024] Determine the observation matrix H i , the observation matrix H i Used to express the relationship between state variables and observations;

[0025] Determine the covariance matrix R of the observation noise i , the covariance matrix R of the observation noise i Used to represent the uncertainty of noise in monitoring data;

[0026] Calculate the Kalman gain K for each data source i , represents the weight of the monitoring data of each data source when updating the state estimation, and the calculation formula is:

[0027] K i =P i-1 ·H i T ·(H i ·P i-1 ·H i T +R i ) -1 ,

[0028] Among them, K i is the Kalman gain, P i-1 is the previous error covariance matrix, H i is the previous observation matrix, R i is the covariance matrix of the previous observation noise, H i T is the previous observation matrix H i The transpose of (H i ·P i-1 ·H i T +R i ) -1 Represents the matrix (H i ·P i-1 ·H i T +R i ) is the inverse matrix of .

[0029] Furthermore, the step of adaptively cleaning the monitoring data includes:

[0030] Build an autoencoder model based on the autoencoder;

[0031] For each data source, the monitoring data is adaptively cleaned by the autoencoder model.

[0032] Furthermore, the step of adaptively cleaning the monitoring data through the autoencoder model includes:

[0033] Perform encoding and decoding operations on the monitoring data through the trained autoencoder model to obtain a decoding result;

[0034] Calculating a reconstruction error of the monitoring data according to the decoding result, wherein the reconstruction error is a difference between the original monitoring data and the decoding result;

[0035] When determining whether the reconstruction error exceeds a preset threshold;

[0036] If it exceeds a preset threshold, it is determined that the monitoring data is an abnormal value or contains noise;

[0037] If the monitoring data is an outlier or contains noise, the monitoring data is cleaned.

[0038] Furthermore, the step of constructing an autoencoder model based on the autoencoder includes:

[0039] Collect historical monitoring data and construct a data set, wherein the data set includes normal monitoring data, abnormal values, and monitoring data containing noise;

[0040] Design a network structure including a target encoder and a target decoder, i.e., an autoencoder, wherein the target encoder is used to compress the input monitoring data into a representation, and the target decoder is used to reconstruct the original input from the compressed representation;

[0041] The autoencoder is trained using the data set to obtain a trained autoencoder model; during training, a reconstruction error is minimized using a back propagation algorithm and an optimizer, where the reconstruction error is the difference between the original monitoring data and the target decoder output.

[0042] The present invention also provides a multi-source monitoring data information fusion processing system, comprising:

[0043] Monitoring data acquisition module: used to obtain monitoring data of the target from different data sources;

[0044] Cleaning module: used for adaptively cleaning the monitoring data;

[0045] Fusion processing module: used to fuse the monitoring data from different data sources through an iterative optimization algorithm to obtain a fusion result.

[0046] In summary, according to the above-mentioned multi-source monitoring data information fusion processing method, the multi-source monitoring data information fusion processing method proposed in the present invention significantly improves the comprehensiveness and accuracy of data processing. First, by acquiring target monitoring data from different data sources to integrate information from multiple perspectives and dimensions, the actual state of the target is fully reflected, and the information loss of a single data source is effectively avoided; secondly, through adaptive cleaning technology, the monitoring data is flexibly and automatically cleaned, which can effectively remove noise and outliers, ensure data quality, and improve the intelligence level and efficiency of data processing; finally, the multi-source monitoring data is fused and processed using an iterative optimization algorithm, which can continuously adjust and optimize the data fusion process, thereby improving the accuracy and stability of the fusion results. This iterative process helps to adapt to the differences and changes between different data sources, ensure the continuous optimization of the fusion results, and thus obtain more accurate and reliable fusion results.

[0047] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0049] Figure 1 This is a flow chart of a multi-source monitoring data information fusion processing method according to a first embodiment of the present invention;

[0050] Figure 2 This is a system block diagram of a multi-source monitoring data information fusion processing system according to a second embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0052] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0054] Example 1

[0055] See also Figure 1 The present invention proposes a multi-source monitoring data information fusion processing method, which includes steps S101 to S103:

[0056] S101, acquiring monitoring data of a target from different data sources.

[0057] It should be noted that obtaining data from different data sources means that the source of information is diversified, which helps to obtain more comprehensive and richer data. For example, for an enterprise, internal data can come from databases, logs, business applications, etc., while external data can come from commercial databases, open data sources or social media.

[0058] However, when acquiring data from different data sources, the quality and accuracy of the data need to be considered. The data quality of different data sources may vary, so the data needs to be verified and cleaned to ensure the reliability of the data. And because the data sources are different, the format, structure, naming conventions, etc. of the data may be different, which brings challenges to data integration. Therefore, it is necessary to consider standardizing the data during the data collection process.

[0059] S102, performing adaptive cleaning on the monitoring data.

[0060] It should be noted that the main purpose of cleaning is to remove erroneous, inaccurate, incomplete, duplicate and redundant information in the data, thereby enhancing the value and utilization of the data. The cleaned data is more credible and can more accurately reflect the actual situation. Adaptive cleaning means that the cleaning process can automatically adjust the cleaning strategy according to the characteristics and quality issues of the data. This flexibility makes the cleaning process more efficient and can cope with data of different types and sources.

[0061] S103, fusing the monitoring data from different data sources through an iterative optimization algorithm to obtain a fusion result.

[0062] It should be noted that by integrating information from different data sources, data from multiple perspectives can be integrated, thereby reducing the errors or deviations that may be caused by a single data source and improving the accuracy and reliability of the data.

[0063] This embodiment can continuously adjust and optimize the data fusion process through an iterative optimization algorithm, thereby improving the accuracy and stability of the fusion result. This iterative process helps to adapt to the differences and changes between different data sources and ensure the continuous optimization of the fusion result.

[0064] In actual applications, data sources may change. Specifically, data sources may fluctuate or become abnormal due to environmental changes, equipment failures, data transmission problems, or other external factors. If the data fusion system cannot detect these changes in real time and adjust its fusion strategy accordingly, the accuracy and reliability of the fusion results may be affected. Therefore, in view of the fact that the prior art cannot dynamically adjust the fusion strategy according to the real-time changes of the data source, the embodiments of the present invention introduce an iterative optimization algorithm to evaluate and adjust the weights and fusion methods of the data sources in real time, dynamically adapt to these changes, ensure the real-time and effectiveness of the fusion results, and thus improve the accuracy and adaptability of the data fusion results.

[0065] Before data fusion, data from different data sources need to be preprocessed, including data cleaning, format conversion, standardization, etc., to ensure data consistency and comparability. And select a suitable iterative optimization algorithm for data fusion, which may involve adjusting and optimizing the parameters of the algorithm to achieve the best fusion effect. During the fusion process, the fusion effect needs to be evaluated regularly, such as by calculating the error between the fusion result and the true value to measure the accuracy of the fusion, so that problems in the fusion process can be discovered and corrected in a timely manner to ensure the reliability of the fusion result.

[0066] Based on step S101 to step S103, the multi-source monitoring data information fusion processing method proposed in this embodiment significantly improves the comprehensiveness and accuracy of data processing. First, by acquiring target monitoring data from different data sources to integrate information from multiple perspectives and dimensions, the actual state of the target is fully reflected, and the information loss of a single data source is effectively avoided; secondly, the monitoring data is flexibly and automatically cleaned through adaptive cleaning technology, which can effectively remove noise and outliers, ensure data quality, and improve the intelligence level and efficiency of data processing; finally, the multi-source monitoring data is fused and processed using an iterative optimization algorithm, which can continuously adjust and optimize the data fusion process, thereby improving the accuracy and stability of the fusion results. This iterative process helps to adapt to the differences and changes between different data sources, ensure the continuous optimization of the fusion results, and thus obtain more accurate and reliable fusion results.

[0067] The following is a detailed introduction to the multi-source monitoring data information fusion processing method according to an embodiment of the present invention.

[0068] Further optionally, in step S103, the step of fusing the monitoring data from different data sources through an iterative optimization algorithm to obtain a fusion result includes:

[0069] Set the initial state and initial error covariance matrix;

[0070] Performing time synchronization and spatial alignment on the monitoring data from different data sources;

[0071] By using an iterative optimization algorithm, combined with the dynamic model and the previous state, the multi-source monitoring data at each time point is fused, and the fusion result at each time point is obtained.

[0072] Understandably, this step is mainly to achieve the effective fusion of multi-source monitoring data to obtain more accurate and comprehensive data results. First, by setting the initial state and the initial error covariance matrix, a starting point and a basis for evaluating the fusion effect are provided for the subsequent iterative optimization algorithm to ensure that the data fusion process can proceed smoothly and converge to the optimal solution. Secondly, the monitoring data from different data sources are synchronized and aligned in time and space to eliminate the differences in time and space caused by different data sources, so that the data of each data source are in the same time and space reference system. Finally, the iterative optimization algorithm is used to combine the dynamic model and the previous state to fuse the multi-source monitoring data at each time point, and through continuous iteration and optimization, the fusion result is more in line with the actual monitoring situation, so as to obtain accurate and reliable fusion results. By setting the initial state, time and space alignment, and using the iterative optimization algorithm for fusion, a more accurate and comprehensive multi-source monitoring data fusion result can be obtained.

[0073] Further optionally, the step of using an iterative optimization algorithm, combining a dynamic model and a previous state, fusing the multi-source monitoring data at each time point, and obtaining a fusion result at each time point includes:

[0074] For each time point, calculate the previous state estimate x i-1 The difference between the monitoring data of each data source and the residual of each data source is obtained, where the previous state estimate x in the first iteration is i-1 is the initial state x0;

[0075] Calculating a Kalman gain for each data source, wherein the Kalman gain reflects the weight of each data source when updating the state estimate;

[0076] The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i ;

[0077] According to the preset convergence threshold and the maximum number of iterations N, the current state of each data source is iteratively updated to obtain the final current state estimation value of each data source;

[0078] The final current state estimation values ​​of all data sources are weighted averaged to obtain the fusion result X of the monitoring data of all data sources.

[0079] Understandably, the deviation between the data source and the model prediction is quantified by accurately calculating the difference between each data source and the previous state estimate, i.e., the residual. Furthermore, by calculating the Kalman gain of each data source, the contribution of different data sources in updating the state estimate, i.e., the weight, is reasonably evaluated. Subsequently, the Kalman gain and residual of these data sources are used to fine-tune and update the current state estimate of each data source to more accurately approximate the true state. The final reliable state estimate of each data source is obtained through iterative optimization. Finally, the state estimates of all data sources are weighted averaged to achieve effective fusion of multi-source monitoring data and generate comprehensive and accurate monitoring results.

[0080] Specifically, in some cases, a data source may start to provide inaccurate data due to failure or interference. If the fusion system cannot recognize this in time and reduce the weight of the data source, the overall fusion result may be misleading. Similarly, when the quality of a data source suddenly improves, the fusion system should be able to increase its weight in real time to take full advantage of this improvement. Therefore, by using the Kalman gain and residual of each data source to fine-tune and update the current state estimate of each data source, the weight and fusion method of the data source can be evaluated and adjusted in real time, thereby further improving the accuracy and adaptability of the data fusion result.

[0081] Further optionally, the current state estimation value x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i The steps include:

[0082] The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i , the calculation formula is: i =x i-1 +K i ·(zH i ·x i-1 ), where z is the monitoring data value of the corresponding data source.

[0083] Understandably, the Kalman gain is used to weigh the contribution of each data source to the state estimation, while taking into account the difference between the actual observation value of the data source and the predicted value based on the previous state estimation, i.e., the residual. In this way, the current state of the system can be reflected more accurately, improving the accuracy and reliability of the state estimation.

[0084] Further optionally, the current state estimation value x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i The steps include:

[0085] Calculate and update the current error covariance matrix P i , the calculation formula is: i =(IK i ·H i )·P i-1 , where I is the identity matrix, P i-1 is the previous error covariance matrix.

[0086] It can be understood that the above formula can evaluate the error propagation of state estimation after integrating the new data source information. This step not only considers the error covariance matrix P of the previous state i-1 , and also through the Kalman gain K i and the observation matrix H i To adjust the propagation of errors, so as to more accurately reflect the reliability of the current state estimation.

[0087] Further optionally, the step of calculating the Kalman gain of each data source includes:

[0088] Determine the observation matrix H i , the observation matrix H i Used to express the relationship between state variables and observations;

[0089] Determine the covariance matrix R of the observation noise i , the covariance matrix R of the observation noise i Used to represent the uncertainty of noise in monitoring data;

[0090] Calculate the Kalman gain K for each data source i , represents the weight of the monitoring data of each data source when updating the state estimation, and the calculation formula is:

[0091] K i =P i-1 ·H i T ·(H i ·P i-1 ·H i T +Ri ) -1 ,

[0092] Among them, K i is the Kalman gain, P i-1 is the previous error covariance matrix, H i is the previous observation matrix, R i is the covariance matrix of the previous observation noise, H i T is the previous observation matrix H i The transpose of (H i ·P i-1 ·H i T +R i ) -1 Represents the matrix (H i ·P i-1 ·H i T +R i ) is the inverse matrix of .

[0093] It is understandable that when there are multiple data sources, each data source will have a corresponding Kalman gain K i , this gain will be adjusted according to the reliability or accuracy of the data source. First, determine the observation matrix H i , in order to clarify the conversion relationship between state variables and observations, which is the basis for understanding how the data source reflects the system state. Next, determine the covariance matrix R of the observation noise i , the impact of noise in the monitoring data can be quantified, so as to more accurately evaluate the reliability of the data. Finally, the Kalman gain K of each data source is obtained by the above calculation formula: i , this gain essentially represents the weight of each data source when updating the state estimation. This step accurately determines the contribution of different data sources to the state estimation by comprehensively considering the observation matrix, the uncertainty of the observation noise, and the error covariance matrix of the previous state, thereby ensuring that the information of each data source can be more reasonably utilized in the data fusion process and improving the accuracy of the state estimation.

[0094] Further optionally, the step of iteratively updating the current state of each data source according to a preset convergence threshold and a maximum number of iterations N to obtain a final current state estimation value of each data source includes:

[0095] The current state estimate x i and the previous state estimate x i-1 The difference between is compared with the convergence threshold;

[0096] If the difference is greater than the preset convergence threshold, the current state estimate xi Set as the starting state for the next iteration;

[0097] After N iterations, the final state estimate x is output after the iteration ends. N as the final state estimate of the corresponding data source;

[0098] If the difference is less than the preset convergence threshold, the iteration is stopped and the updated current state estimate x is i As a state estimate of the corresponding data source.

[0099] It is understandable that by continuously comparing the difference between the current state estimate and the previous state estimate, it is determined whether the state estimate has reached the preset convergence condition. If the difference is greater than the convergence threshold, the iteration continues, and the current state is used as the starting point for the next iteration until the difference is reduced to below the threshold or the maximum number of iterations N is reached. This process ensures the accuracy of the state estimate and the convergence of the algorithm, and the final output is a fully optimized state estimate. Whether the iteration is terminated early due to reaching the convergence condition or terminated after completing N iterations, the final reliable state estimate of the corresponding data source can be obtained.

[0100] Further optionally, in step S102, the step of adaptively cleaning the monitoring data includes:

[0101] Build an autoencoder model based on the autoencoder;

[0102] For each data source, the monitoring data is adaptively cleaned by the autoencoder model.

[0103] Understandably, the monitoring data from different data sources are adaptively cleaned through the autoencoder model to improve the data quality.

[0104] Further optionally, the step of adaptively cleaning the monitoring data by using the autoencoder model includes:

[0105] Perform encoding and decoding operations on the monitoring data through the trained autoencoder model to obtain a decoding result;

[0106] Calculating a reconstruction error of the monitoring data according to the decoding result, wherein the reconstruction error is a difference between the original monitoring data and the decoding result;

[0107] When determining whether the reconstruction error exceeds a preset threshold;

[0108] If it exceeds a preset threshold, it is determined that the monitoring data is an abnormal value or contains noise;

[0109] If the monitoring data is an outlier or contains noise, the monitoring data is cleaned.

[0110] It is understandable that adaptive cleaning of monitoring data through the autoencoder model can improve data quality. Specifically, first, the monitoring data is encoded and decoded using the trained autoencoder model to obtain the decoding result; second, the difference between the original monitoring data and the decoding result, i.e., the reconstruction error, is calculated based on the decoding result; then, outliers or noise data are identified by judging whether the reconstruction error exceeds the preset threshold; finally, the monitoring data that is judged to be abnormal or contains noise is cleaned. This process can accurately identify and remove bad information in the data, ensure the accuracy and reliability of the data, and provide a high-quality data set for subsequent data processing and analysis.

[0111] Further optionally, the step of constructing an autoencoder model based on the autoencoder includes:

[0112] Collect historical monitoring data and construct a data set, wherein the data set includes normal monitoring data, abnormal values, and monitoring data containing noise;

[0113] Design a network structure including a target encoder and a target decoder, i.e., an autoencoder, wherein the target encoder is used to compress the input monitoring data into a representation, and the target decoder is used to reconstruct the original input from the compressed representation;

[0114] The autoencoder is trained using the data set to obtain a trained autoencoder model; during training, a reconstruction error is minimized using a back propagation algorithm and an optimizer, where the reconstruction error is the difference between the original monitoring data and the target decoder output.

[0115] Understandably, an autoencoder model is constructed for adaptive cleaning of monitoring data. Specifically, first, historical monitoring data containing normal monitoring data, outliers, and noisy data are collected to construct a comprehensive data set; second, a network structure of an autoencoder is designed, which consists of a target encoder and a target decoder, which are used to compress the input data into a low-dimensional representation and reconstruct the original input from the low-dimensional representation; finally, the autoencoder is trained using the collected data set through the back-propagation algorithm and the optimizer, and the model parameters are optimized by minimizing the difference between the original monitoring data and the decoder output, that is, the reconstruction error, and finally a trained autoencoder model is obtained. This process ensures that the autoencoder can accurately learn the feature differences between normal data and abnormal data, thereby achieving adaptive cleaning of monitoring data.

[0116] Example 2

[0117] See also Figure 2The present invention also proposes a multi-source monitoring data information fusion processing system, which includes:

[0118] Monitoring data acquisition module: used to obtain monitoring data of the target from different data sources;

[0119] Cleaning module: used for adaptively cleaning the monitoring data;

[0120] Fusion processing module: used to fuse the monitoring data from different data sources through an iterative optimization algorithm to obtain a fusion result.

[0121] Further optionally, the fusion processing module is also used for:

[0122] Set the initial state and initial error covariance matrix;

[0123] Performing time synchronization and spatial alignment on the monitoring data from different data sources;

[0124] By using an iterative optimization algorithm, combined with the dynamic model and the previous state, the multi-source monitoring data at each time point is fused, and the fusion result at each time point is obtained.

[0125] Further optionally, the fusion processing module is also used for:

[0126] For each time point, calculate the previous state estimate x i-1 The difference between the monitoring data of each data source and the residual of each data source is obtained, where the previous state estimate x in the first iteration is i-1 is the initial state x0;

[0127] Calculating a Kalman gain for each data source, wherein the Kalman gain reflects the weight of each data source when updating the state estimate;

[0128] The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i ;

[0129] According to the preset convergence threshold and the maximum number of iterations N, the current state of each data source is iteratively updated to obtain the final current state estimation value of each data source;

[0130] The final current state estimation values ​​of all data sources are weighted averaged to obtain the fusion result X of the monitoring data of all data sources.

[0131] Further optionally, the fusion processing module is also used for:

[0132] The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i, the calculation formula is: i =x i-1 +K i ·(zH i ·x i-1 ), where z is the monitoring data value of the corresponding data source.

[0133] Further optionally, the fusion processing module is also used for:

[0134] Calculate and update the current error covariance matrix P i , the calculation formula is: i =(IK i ·H i )·P i-1 , where I is the identity matrix, P i-1 is the previous error covariance matrix.

[0135] Further optionally, the fusion processing module is also used for:

[0136] Determine the observation matrix H i , the observation matrix H i Used to express the relationship between state variables and observations;

[0137] Determine the covariance matrix R of the observation noise i , the covariance matrix R of the observation noise i Used to represent the uncertainty of noise in monitoring data;

[0138] Calculate the Kalman gain K for each data source i , represents the weight of the monitoring data of each data source when updating the state estimation, and the calculation formula is:

[0139] K i =P i-1 ·H i T ·(H i ·P i-1 ·H i T +R i ) -1 ,

[0140] Among them, K i is the Kalman gain, P i-1 is the previous error covariance matrix, H i is the previous observation matrix, R i is the covariance matrix of the previous observation noise, H i T is the previous observation matrix H i The transpose of (H i ·P i-1 ·Hi T +R i ) -1 Represents the matrix (H i ·P i-1 ·H i T +R i ) is the inverse matrix of .

[0141] Further optionally, the cleaning module is also used for:

[0142] Build an autoencoder model based on the autoencoder;

[0143] For each data source, the monitoring data is adaptively cleaned by the autoencoder model.

[0144] Further optionally, the cleaning module is also used for:

[0145] Perform encoding and decoding operations on the monitoring data through the trained autoencoder model to obtain a decoding result;

[0146] Calculating a reconstruction error of the monitoring data according to the decoding result, wherein the reconstruction error is a difference between the original monitoring data and the decoding result;

[0147] When determining whether the reconstruction error exceeds a preset threshold;

[0148] If it exceeds a preset threshold, it is determined that the monitoring data is an abnormal value or contains noise;

[0149] If the monitoring data is an outlier or contains noise, the monitoring data is cleaned.

[0150] Further optionally, the cleaning module is also used for:

[0151] Collect historical monitoring data and construct a data set, wherein the data set includes normal monitoring data, abnormal values, and monitoring data containing noise;

[0152] Design a network structure including a target encoder and a target decoder, i.e., an autoencoder, wherein the target encoder is used to compress the input monitoring data into a representation, and the target decoder is used to reconstruct the original input from the compressed representation;

[0153] The autoencoder is trained using the data set to obtain a trained autoencoder model; during training, a reconstruction error is minimized using a back propagation algorithm and an optimizer, where the reconstruction error is the difference between the original monitoring data and the target decoder output.

[0154] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-source monitoring data information fusion processing method, characterized in that: The method comprises: Obtain monitoring data of the target from different data sources; Adaptively cleaning the monitoring data; The monitoring data from different data sources are fused through an iterative optimization algorithm to obtain a fusion result.

2. The multi-source monitoring data information fusion processing method according to claim 1 is characterized in that: The step of fusing the monitoring data from different data sources through an iterative optimization algorithm to obtain a fusion result comprises: Set the initial state and initial error covariance matrix; Performing time synchronization and spatial alignment on the monitoring data from different data sources; By using an iterative optimization algorithm, combined with the dynamic model and the previous state, the multi-source monitoring data at each time point is fused, and the fusion result at each time point is obtained.

3. The multi-source monitoring data information fusion processing method according to claim 2 is characterized in that: The step of using an iterative optimization algorithm, combining a dynamic model and a previous state, fusing the multi-source monitoring data at each time point, and obtaining a fusion result at each time point includes: For each time point, calculate the previous state estimate x i-1 The difference between the monitoring data of each data source and the residual of each data source is obtained, where the previous state estimate x in the first iteration is i-1 is the initial state x0; Calculating a Kalman gain for each data source, wherein the Kalman gain reflects the weight of each data source when updating the state estimate; The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i ; According to the preset convergence threshold and the maximum number of iterations N, the current state of each data source is iteratively updated to obtain the final current state estimation value of each data source; The final current state estimation values ​​of all data sources are weighted averaged to obtain the fusion result X of the monitoring data of all data sources.

4. The multi-source monitoring data information fusion processing method according to claim 3 is characterized in that: The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i The steps include: The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i , the calculation formula is: i =x i-1 +K i ·(zH i ·x i-1 ), where z is the monitoring data value of the corresponding data source.

5. The multi-source monitoring data information fusion processing method according to claim 3 is characterized in that: The current state estimate x of each data source is calculated and updated according to the Kalman gain and residual of each data source. i The steps include: Calculate and update the current error covariance matrix P i , the calculation formula is: i =(IK i ·H i )·P i-1 , where I is the identity matrix, P i-1 is the previous error covariance matrix.

6. The multi-source monitoring data information fusion processing method according to claim 3 is characterized in that: The step of calculating the Kalman gain of each data source comprises: Determine the observation matrix H i , the observation matrix H i Used to express the relationship between state variables and observations; Determine the covariance matrix R of the observation noise i , the covariance matrix R of the observation noise i Used to represent the uncertainty of noise in monitoring data; Calculate the Kalman gain K for each data source i , represents the weight of the monitoring data of each data source when updating the state estimation, and the calculation formula is: K i =P i-1 ·H i T ·(H i ·P i-1 ·H i T +R i ) -1 , Among them, K i is the Kalman gain, P i-1 is the previous error covariance matrix, H i is the previous observation matrix, R i is the covariance matrix of the previous observation noise, H i T is the previous observation matrix H i The transpose of (H i ·P i-1 ·H i T +R i ) -1 Represents the matrix (H i ·P i-1 ·H i T +R i ) is the inverse matrix of .

7. The multi-source monitoring data information fusion processing method according to claim 1 is characterized in that: The step of adaptively cleaning the monitoring data comprises: Build an autoencoder model based on the autoencoder; For each data source, the monitoring data is adaptively cleaned by the autoencoder model.

8. The multi-source monitoring data information fusion processing method according to claim 7 is characterized in that: The step of adaptively cleaning the monitoring data by the autoencoder model comprises: Perform encoding and decoding operations on the monitoring data through the trained autoencoder model to obtain a decoding result; Calculating a reconstruction error of the monitoring data according to the decoding result, wherein the reconstruction error is a difference between the original monitoring data and the decoding result; When determining whether the reconstruction error exceeds a preset threshold; If it exceeds a preset threshold, it is determined that the monitoring data is an abnormal value or contains noise; If the monitoring data is an outlier or contains noise, the monitoring data is cleaned.

9. The multi-source monitoring data information fusion processing method according to claim 7 is characterized in that: The step of constructing an autoencoder model based on the autoencoder includes: Collect historical monitoring data and construct a data set, wherein the data set includes normal monitoring data, abnormal values, and monitoring data containing noise; Design a network structure including a target encoder and a target decoder, i.e., an autoencoder, wherein the target encoder is used to compress the input monitoring data into a representation, and the target decoder is used to reconstruct the original input from the compressed representation; The autoencoder is trained using the data set to obtain a trained autoencoder model; during training, a reconstruction error is minimized using a back propagation algorithm and an optimizer, where the reconstruction error is the difference between the original monitoring data and the target decoder output.

10. The multi-source monitoring data information fusion processing system is characterized by: include: Monitoring data acquisition module: used to obtain monitoring data of the target from different data sources; Cleaning module: used for adaptively cleaning the monitoring data; Fusion processing module: used to fuse the monitoring data from different data sources through an iterative optimization algorithm to obtain a fusion result.

Citation Information

Patent Citations

  • Improved data cleaning method for stack noise reduction auto-encoder

    CN109978079A

  • Electric energy quality data cleaning method and device based on Spark

    CN111552685A

  • Method for identifying traffic lights by fusing multi-source data

    CN117037525A

  • Abnormity detection method and system based on improved noise reduction auto-encoder

    CN117688305A

  • Substation communication network equipment monitoring and fault early warning method and system

    CN118200118A

Cited By

  • Multi-source heterogeneous agricultural data real-time cleaning method and system

    CN120670727A

  • Intelligent visualization system for dynamic aggregation of big data

    CN120705376A

  • Safety monitoring data fusion system and method based on adaptive Kalman filtering

    CN121524970A