Multi-source data fusion method and system for heterogeneous data sources

Abstract

The invention is suitable for the technical field of multi-source data fusion, and provides a multi-source data fusion method and system for heterogeneous data sources. The method comprises the following steps: receiving a data fusion request, and collecting a plurality of heterogeneous collection data; defect identification is carried out, and fusion defect time and a defect data source are determined; matching a plurality of homologous collection data and a plurality of heterogenous simultaneous data; determining a grade estimation time period, performing defect estimation, and calculating target defect data; performing contrast verification on the target defect data, and when the verification is not passed, expanding the grade estimation time period step by step until the verification is passed; and carrying out multi-source data fusion processing. The method can determine fusion defect time and a defect data source, determine a grade estimation time period, calculate target defect data, perform contrast verification, and expand the grade estimation time period step by step until verification is passed when verification is not passed, so that automatic identification and accurate supplement of data defects can be realized, the comprehensiveness of data fusion is improved, and the data fusion efficiency is improved. And the multi-source data fusion effect is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-source data fusion, and particularly relates to a multi-source data fusion method and system for heterogeneous data sources. Background Art

[0002] Multi-source data fusion is a technology that integrates data from different sources, formats, and structures to obtain more comprehensive and accurate information, and provides better support for decision-making, prediction, and analysis. Multi-source data fusion technology has been widely applied in multiple fields, including intelligent transportation, healthcare, industrial production, image detection, and environmental monitoring, etc.

[0003] In multi-source data fusion technology, the collection of multi-source data is particularly important.

[0004] In the prior art, for the collection of multi-source data, it is impossible to identify and detect data defects according to the requirements of multi-source data fusion, and it is impossible to perform precise defect supplementation according to actual data defects, resulting in incomplete fusion data and affecting the effect of multi-source data fusion. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a multi-source data fusion method and system for heterogeneous data sources, aiming to solve the technical problems existing in the prior art mentioned in the background art.

[0006] The embodiments of the present invention are implemented as follows:

[0007] A multi-source data fusion method for heterogeneous data sources, the method specifically includes the following steps:

[0008] Receive a data fusion request, determine the fusion period frequency and multiple heterogeneous data sources, and collect multiple heterogeneous acquisition data from the multiple heterogeneous data sources according to the fusion period frequency;

[0009] Based on the fusion period frequency, perform defect identification on the multiple heterogeneous acquisition data to determine the fusion defect time and the defect data source;

[0010] According to the fusion defect time and the defect data source, match multiple homologous acquisition data and multiple heterologous simultaneous data from the multiple heterogeneous acquisition data;

[0011] Determine the level estimation period, select multiple level period data from the multiple homologous acquisition data for defect estimation, and calculate the target defect data;

[0012] Based on the multiple heterologous simultaneous data, perform control verification on the target defect data, and when the verification fails, gradually expand the level estimation period until the verification passes;

[0013] Perform multi-source data fusion processing.

[0014] As a further limitation of the technical solution of the embodiment of the present invention, the steps of receiving a data fusion request, determining the fusion period frequency and multiple heterogeneous data sources, and collecting multiple heterogeneous acquisition data from the multiple heterogeneous data sources according to the fusion period frequency specifically include the following steps:

[0015] Receive a data fusion request;

[0016] Identify the data fusion request to determine the fusion period frequency and multiple heterogeneous data sources;

[0017] Determine the data acquisition period according to the fusion period frequency;

[0018] Create a data acquisition task according to the data acquisition period and the multiple heterogeneous data sources;

[0019] Collect multiple heterogeneous acquisition data according to the data acquisition task.

[0020] As a further limitation of the technical solution of the embodiment of the present invention, the steps of performing defect identification on the multiple heterogeneous acquisition data based on the fusion period frequency to determine the fusion defect time and the defect data source specifically include the following steps:

[0021] Determine the fusion frequency time sequence according to the fusion period frequency;

[0022] Screen multiple frequency time sequence data from the multiple heterogeneous acquisition data according to the fusion frequency time sequence;

[0023] Perform defect identification on the multiple frequency time sequence data based on the fusion frequency time sequence to determine the fusion defect time and the defect data source.

[0024] As a further limitation of the technical solution of the embodiment of the present invention, the steps of matching multiple homologous acquisition data and multiple heterologous simultaneous data from the multiple heterogeneous acquisition data according to the fusion defect time and the defect data source specifically include the following steps:

[0025] Screen multiple homologous acquisition data from the multiple frequency time sequence data according to the defect data source;

[0026] Match multiple heterologous simultaneous data from the multiple frequency time sequence data according to the fusion defect time.

[0027] As a further limitation of the technical solution of the embodiment of the present invention, the steps of determining the grade estimation period, selecting multiple grade period data from the multiple homologous acquisition data for defect estimation, and calculating the target defect data specifically include the following steps:

[0028] Determine the level prediction period;

[0029] Taking the fusion defect time as the origin, divide the level prediction period into a pre-level prediction period and a post-level prediction period;

[0030] According to the pre-level prediction period and the post-level prediction period, select multiple level period data from multiple pieces of the homologous collected data;

[0031] Based on multiple pieces of the level period data, perform defect prediction and calculate the target defect data.

[0032] As a further limitation of the technical solution of the embodiment of the present invention, the calculation formula of the target defect data is:

[0033]

[0034] where j represents the j-th level prediction period, B j is the target defect data of the j-th level prediction period, -n j is the pre-level prediction period of the j-th level prediction period, n j is the post-level prediction period of the j-th level prediction period, is the level period data at the i-th time of the j-th level prediction period, is the period length of the i-th time of the j-th level prediction period, and k is a preset period length decomposition factor.

[0035] As a further limitation of the technical solution of the embodiment of the present invention, the step of comparing and verifying the target defect data based on multiple pieces of the heterologous simultaneous data and, when the verification fails, gradually expanding the level prediction period specifically includes the following steps:

[0036] Analyze multiple pieces of the heterologous simultaneous data to construct a standard verification interval;

[0037] According to the standard verification interval, compare and verify the target defect data to determine whether the verification passes;

[0038] When the verification fails, generate a level expansion instruction;

[0039] According to the level expansion instruction, gradually expand the level prediction period.

[0040] As a further limitation of the technical solution of the embodiment of the present invention, the step of performing multi-source data fusion processing specifically includes the following steps:

[0041] When the verification passes, determine the effective defect data;

[0042] Supplement the valid defect data in the multiple frequency time series data, and perform data collation to generate a complete time series data set;

[0043] Perform multi-source data fusion on the complete time series data set to generate multi-source fusion data.

[0044] A multi-source data fusion system for heterogeneous data sources, the system includes a multi-source data acquisition module, a data defect identification module, a data matching processing module, a defect prediction calculation module, a control verification processing module, and a multi-source data fusion module, where:

[0045] The multi-source data acquisition module is used to receive a data fusion request, determine the fusion period frequency and multiple heterogeneous data sources, and collect multiple heterogeneous acquisition data from the multiple heterogeneous data sources according to the fusion period frequency;

[0046] The data defect identification module is used to identify defects in the multiple heterogeneous acquisition data based on the fusion period frequency, and determine the fusion defect time and the defect data source;

[0047] The data matching processing module is used to match multiple homologous acquisition data and multiple heterologous simultaneous data from the multiple heterogeneous acquisition data according to the fusion defect time and the defect data source;

[0048] The defect prediction calculation module is used to determine the level prediction period, select multiple level period data from the multiple homologous acquisition data for defect prediction, and calculate the target defect data;

[0049] The control verification processing module is used to perform control verification on the target defect data based on the multiple heterologous simultaneous data, and when the verification fails, gradually expand the level prediction period until the verification passes;

[0050] The multi-source data fusion module is used to perform multi-source data fusion processing.

[0051] As a further limitation of the technical solution of the embodiment of the present invention, the control verification processing module specifically includes:

[0052] The interval construction unit is used to analyze the multiple heterologous simultaneous data and construct a standard verification interval;

[0053] The control verification unit is used to perform control verification on the target defect data according to the standard verification interval to determine whether the verification passes;

[0054] The instruction generation unit is used to generate a level expansion instruction when the verification fails;

[0055] A step-by-step expansion unit for step-by-step expanding the level estimation period according to the level expansion instruction.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] In the embodiment of the present invention, by receiving a data fusion request, multiple heterogeneous acquisition data are collected; defect identification is performed to determine the fusion defect time and defect data sources; multiple homologous acquisition data and multiple heterologous simultaneous data are matched; a level estimation period is determined, defect prediction is performed, and target defect data is calculated; the target defect data is subjected to control verification, and when the verification fails, the level estimation period is expanded step by step until the verification passes; multi-source data fusion processing is performed. It is possible to determine the fusion defect time and defect data sources, determine the level estimation period, calculate the target defect data, and perform control verification. When the verification fails, the level estimation period is expanded step by step until the verification passes, which can realize the automatic identification and precise supplementation of data defects, improve the comprehensiveness of data fusion, and ensure the effect of multi-source data fusion. Description of the Drawings

[0058] Figure 1 Shows a flowchart of a multi-source data fusion method for heterogeneous data sources provided by an embodiment of the present invention;

[0059] Figure 2 Shows a flowchart of collecting multiple heterogeneous acquisition data in the method provided by an embodiment of the present invention;

[0060] Figure 3 Shows a flowchart of determining the fusion defect time and defect data sources in the method provided by an embodiment of the present invention;

[0061] Figure 4 Shows a flowchart of matching multiple homologous acquisition data and multiple heterologous simultaneous data in the method provided by an embodiment of the present invention;

[0062] Figure 5 Shows a flowchart of calculating target defect data in the method provided by an embodiment of the present invention;

[0063] Figure 6 Shows a flowchart of performing control verification and processing in the method provided by an embodiment of the present invention;

[0064] Figure 7 Shows a flowchart of multi-source data fusion processing in the method provided by an embodiment of the present invention;

[0065] Figure 8 Shows an application architecture diagram of a multi-source data fusion system for heterogeneous data sources provided by an embodiment of the present invention;

[0066] Figure 9The structural block diagram of the control verification processing module in the system provided by the embodiment of the present invention is shown. Detailed implementation manners

[0067] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0068] It can be understood that in multi-source data fusion technology, the collection of multi-source data is particularly important. In the prior art, for the collection of multi-source data, it is impossible to identify and detect data defects according to the requirements of multi-source data fusion, and it is impossible to accurately supplement defects according to actual data defects, resulting in incomplete fused data and affecting the effect of multi-source data fusion.

[0069] To solve the above problems, a multi-source data fusion method and system for heterogeneous data sources disclosed in an embodiment of the present invention receive a data fusion request, determine a fusion time period frequency and a plurality of heterogeneous data sources, and collect a plurality of heterogeneous collected data from the plurality of heterogeneous data sources according to the fusion time period frequency; based on the fusion time period frequency, perform defect identification on the plurality of heterogeneous collected data to determine a fusion defect time and a defect data source; according to the fusion defect time and the defect data source, match a plurality of homologous collected data and a plurality of heterologous simultaneous data from the plurality of heterogeneous collected data; determine a level estimation time period, select a plurality of level time period data from the plurality of homologous collected data for defect estimation, and calculate target defect data; perform control verification on the target defect data based on the plurality of heterologous simultaneous data, and when the verification fails, gradually expand the level estimation time period until the verification passes; perform multi-source data fusion processing. It can determine the fusion defect time and the defect data source, determine the level estimation time period, calculate the target defect data, and perform control verification. When the verification fails, gradually expand the level estimation time period until the verification passes, which can realize the automatic identification and accurate supplement of data defects, improve the comprehensiveness of data fusion, and ensure the effect of multi-source data fusion.

[0070] Specifically, Figure 1 The flowchart of the multi-source data fusion method for heterogeneous data sources provided by the embodiment of the present invention is shown.

[0071] In a preferred embodiment provided by the present invention, a multi-source data fusion method for heterogeneous data sources, the method specifically includes the following steps:

[0072] Step S101, receive a data fusion request, determine a fusion time period frequency and a plurality of heterogeneous data sources, and collect a plurality of heterogeneous collected data from the plurality of heterogeneous data sources according to the fusion time period frequency.

[0073] In an embodiment of the present invention, when a user has a need for multi-source data fusion, the user can edit and input a data fusion request. By receiving the data fusion request, the system then identifies the data fusion request to determine the fusion period frequency and multiple heterogeneous data sources. From the fusion period frequency, the data collection period is extracted. Then, based on the data collection period and the multiple heterogeneous data sources, a data collection task is created, and further the data collection task is executed. During the data collection period, multiple heterogeneous collection data are collected from the multiple heterogeneous data sources.

[0074] It can be understood that the fusion period frequency is composed of the data collection period and the fusion frequency time sequence. Specifically, the data collection period is a continuous period, for example: from 8:00 to 10:00; the fusion frequency time sequence is composed of multiple punctual times arranged according to frequency and time sequence, for example: 8:00, 8:10, 8:20, ……, where each 10 minutes is a frequency and is arranged according to the time sequence.

[0075] Specifically, Figure 2 shows a flowchart of collecting multiple heterogeneous collection data in the method provided by the embodiment of the present invention.

[0076] Among them, in another preferred embodiment provided by the present invention, the steps of receiving the data fusion request, determining the fusion period frequency and multiple heterogeneous data sources, and collecting multiple heterogeneous collection data from the multiple heterogeneous data sources according to the fusion period frequency specifically include the following steps:

[0077] Step S1011: Receive the data fusion request.

[0078] Step S1012: Identify the data fusion request to determine the fusion period frequency and multiple heterogeneous data sources.

[0079] Step S1013: Determine the data collection period according to the fusion period frequency.

[0080] Step S1014: Create a data collection task according to the data collection period and the multiple heterogeneous data sources.

[0081] Step S1015: Collect multiple heterogeneous collection data according to the data collection task.

[0082] Furthermore, the multi-source data fusion method for the heterogeneous data sources further includes the following steps:

[0083] Step S102: Based on the fusion period frequency, perform defect identification on the multiple heterogeneous collection data to determine the fusion defect time and the defective data source.

[0084] In an embodiment of the present invention, from the fusion period frequencies, a fusion frequency time series is extracted, and then according to multiple times in the fusion frequency time series, multiple frequency time series data are screened from multiple heterogeneous acquisition data, and then based on the multiple times in the fusion frequency time series, defect identification is performed on the multiple frequency time series data to determine the fusion defect time and the defect data source.

[0085] It can be understood that in defect identification, according to multiple times in the fusion frequency time series, identification is performed on multiple frequency time series data. If the data of a certain heterogeneous data source is missing at a certain time, then it is determined that this time is the fusion defect time, and this heterogeneous data source is the defect data source.

[0086] Specifically, Figure 3 The flowchart shows the determination of the fusion defect time and the defect data source in the method provided by the embodiment of the present invention.

[0087] Among them, in another preferred embodiment provided by the present invention, the defect identification of multiple heterogeneous acquisition data based on the fusion period frequencies to determine the fusion defect time and the defect data source specifically includes the following steps:

[0088] Step S1021: Determine the fusion frequency time series according to the fusion period frequencies.

[0089] Step S1022: Screen multiple frequency time series data from multiple heterogeneous acquisition data according to the fusion frequency time series.

[0090] Step S1023: Perform defect identification on multiple frequency time series data based on the fusion frequency time series to determine the fusion defect time and the defect data source.

[0091] Furthermore, the multi-source data fusion method for the heterogeneous data source further includes the following steps:

[0092] Step S103: Match multiple homologous acquisition data and multiple heterologous simultaneous data from multiple heterogeneous acquisition data according to the fusion defect time and the defect data source.

[0093] In an embodiment of the present invention, multiple homologous acquisition data collected from the defect data source are screened from multiple frequency time series data, and multiple heterologous simultaneous data with the acquisition time being the fusion defect time are matched from multiple frequency time series data.

[0094] It can be understood that matching multiple homologous acquisition data and multiple heterologous simultaneous data can be to screen multiple homologous acquisition data belonging to the defect data source and screen multiple heterologous simultaneous data belonging to the fusion defect time from multiple frequency time series data through hash matching, entity matching or combined matching techniques.

[0095] Specifically, Figure 4 The flowchart shows the method provided by the embodiment of the present invention for matching multiple homologous acquisition data and multiple heterologous simultaneous data.

[0096] Wherein, in another preferred embodiment provided by the present invention, the matching of multiple homologous acquisition data and multiple heterologous simultaneous data from multiple heterogeneous acquisition data according to the fusion defect time and the defect data source specifically includes the following steps:

[0097] Step S1031: Screen multiple homologous acquisition data from multiple frequency-time series data according to the defect data source.

[0098] Step S1032: Match multiple heterologous simultaneous data from multiple frequency-time series data according to the fusion defect time.

[0099] Furthermore, the multi-source data fusion method for the heterogeneous data source further includes the following steps:

[0100] Step S104: Determine the level estimation period, select multiple level period data from multiple homologous acquisition data for defect estimation, and calculate the target defect data.

[0101] In the embodiment of the present invention, by determining the level estimation period, and taking the fusion defect time as the origin, the level estimation period is divided into two symmetric segments. The period before the origin is the pre-level estimation period, and the period after the origin is the post-level estimation period. Select multiple level period data whose acquisition times belong to the pre-level estimation period or the post-level estimation period from multiple homologous acquisition data. Then, based on multiple level period data, defect estimation is performed on the data not acquired by the defect data source at the fusion defect time, and the target defect data is calculated. Specifically, the calculation formula for the target defect data is:

[0102]

[0103] Where j represents the j-th level estimation period, B j is the target defect data of the j-th level estimation period, -n j is the pre-level estimation period of the j-th level estimation period, n j is the post-level estimation period of the j-th level estimation period, is the level period data at time i of the j-th level estimation period, is the period length of the j-th level estimation period at time i, and k is a preset period length decomposition factor.

[0104] It can be understood that during the level estimation period, different lengths of time periods are set according to different levels, and as the level increases, the time period is correspondingly doubled. For example, the first-level estimation time period is 0.5h, the second-level estimation time period is 1h, the third-level estimation time period is 1.5h, and so on.

[0105] It can be understood that during the initial defect estimation when calculating the target defect data, the level estimation time period is the first-level estimation time period.

[0106] It can be understood that in different application scenarios, the types of target defect data are different. For example, in industrial production, the target defect data can be loss size, unqualified ratio, etc.; in image detection, the target defect data can be color value, brightness value, etc.; in environmental monitoring, the target defect data can be temperature, carbon dioxide concentration, PM2.5 concentration, wind speed, etc.

[0107] Specifically, Figure 5 The flowchart of calculating the target defect data in the method provided by the embodiment of the present invention is shown.

[0108] Among them, in another preferred embodiment provided by the present invention, when determining the level estimation time period, multiple level time period data are selected from multiple pieces of the homologous acquisition data for defect estimation, and calculating the target defect data specifically includes the following steps:

[0109] Step S1041: Determine the level estimation time period.

[0110] Step S1042: Taking the fusion defect time as the origin, divide the level estimation time period into a pre-level estimation time period and a post-level estimation time period.

[0111] Step S1043: According to the pre-level estimation time period and the post-level estimation time period, select multiple level time period data from multiple pieces of the homologous acquisition data.

[0112] Step S1044: Based on multiple pieces of the level time period data, perform defect estimation and calculate the target defect data.

[0113] Furthermore, the multi-source data fusion method for the heterogeneous data sources further includes the following steps:

[0114] Step S105: Based on multiple pieces of the heterogeneous simultaneous data, perform control verification on the target defect data, and when the verification fails, gradually expand the level estimation time period until the verification passes.

[0115] In an embodiment of the present invention, according to multiple heterologous simultaneous data, a control analysis is performed on target defect data to construct a standard verification interval for the target defect data. Then, based on the standard verification interval, a control verification is performed on the target defect data to determine whether the verification is passed. Specifically, if the target defect data is within the standard verification interval, it is determined that the verification is passed; if the target defect data is not within the standard verification interval, it is determined that the verification fails. At this time, a level expansion instruction is generated, and according to the level expansion instruction, the level prediction period is gradually expanded. With the expanded level prediction period, multiple level period data are reselected for defect prediction, and new target defect data is calculated.

[0116] It can be understood that the control analysis is to analyze the interval range of the target defect data according to multiple heterologous simultaneous data. Although the multiple heterologous simultaneous data and the target defect data are different types of data, there is a correlation between the data. The standard verification interval of the target defect data can be determined by performing a correlation analysis on the multiple heterologous simultaneous data. For example, when the target defect data is a numerical value and the heterologous simultaneous data is a related curve graph, the numerical range of the target defect data can be determined by analyzing the curve graph to obtain the standard verification interval.

[0117] It can be understood that with the expanded level prediction period, more level period data are selected. Therefore, when performing defect prediction, it has a higher accuracy.

[0118] Specifically, Figure 6 The flowchart of performing control verification and processing in the method provided by the embodiment of the present invention is shown.

[0119] Among them, in another preferred embodiment provided by the present invention, the control verification of the target defect data based on the multiple heterologous simultaneous data and the step of gradually expanding the level prediction period when the verification fails specifically include the following steps:

[0120] Step S1051: Analyze the multiple heterologous simultaneous data to construct a standard verification interval.

[0121] Step S1052: According to the standard verification interval, perform a control verification on the target defect data to determine whether the verification is passed.

[0122] Step S1053: Generate a level expansion instruction when the verification fails.

[0123] Step S1054: Gradually expand the level prediction period according to the level expansion instruction.

[0124] Furthermore, the multi-source data fusion method for the heterogeneous data sources further includes the following steps:

[0125] Step S106: Perform multi-source data fusion processing.

[0126] In the embodiment of the present invention, after passing the verification, valid defect data is determined. Then, among multiple frequency-time series data, according to the time series of the valid defect data, the valid defect data is inserted and supplemented, and then data collation is performed to obtain a complete time series data set. Furthermore, multi-source data fusion is performed on the complete time series data set to generate multi-source fusion data.

[0127] It can be understood that for multi-source data fusion, methods such as feature-level fusion and decision-level fusion can be used to perform multi-source data fusion processing on the complete time series data set. Among them, feature-level fusion is to extract the features of the data and then fuse the features; decision-level fusion is to perform data processing and analysis and then fuse based on rules or models.

[0128] Specifically, Figure 7 The flowchart of multi-source data fusion processing in the method provided by the embodiment of the present invention is shown.

[0129] Among them, in another preferred embodiment provided by the present invention, the performing multi-source data fusion processing specifically includes the following steps:

[0130] Step S1061: When the verification passes, determine valid defect data.

[0131] Step S1062: In the multiple frequency-time series data, supplement the valid defect data and perform data collation to generate a complete time series data set.

[0132] Step S1063: Perform multi-source data fusion on the complete time series data set to generate multi-source fusion data.

[0133] Furthermore, Figure 8 The application architecture diagram of the multi-source data fusion system for heterogeneous data sources provided by the embodiment of the present invention is shown.

[0134] Specifically, in another preferred embodiment provided by the present invention, a multi-source data fusion system for heterogeneous data sources includes:

[0135] A multi-source data acquisition module 101, configured to receive a data fusion request, determine a fusion period frequency and multiple heterogeneous data sources, and collect multiple heterogeneous acquisition data from the multiple heterogeneous data sources according to the fusion period frequency.

[0136] In an embodiment of the present invention, when a user has a need for multi-source data fusion, the user can edit and input a data fusion request. The multi-source data acquisition module 101 receives the data fusion request, identifies the fusion time period frequency and multiple heterogeneous data sources, extracts the data acquisition time period from the fusion time period frequency, creates a data acquisition task according to the data acquisition time period and the multiple heterogeneous data sources, and then executes the data acquisition task. During the data acquisition time period, multiple heterogeneous acquisition data are acquired from the multiple heterogeneous data sources.

[0137] The data defect identification module 102 is used to identify defects in the multiple heterogeneous acquisition data based on the fusion time period frequency, and determine the fusion defect time and the defective data source.

[0138] In an embodiment of the present invention, the data defect identification module 102 extracts the fusion frequency time sequence from the fusion time period frequency, then screens multiple frequency time sequence data from the multiple heterogeneous acquisition data according to the multiple times in the fusion frequency time sequence, and then identifies defects in the multiple frequency time sequence data based on the multiple times in the fusion frequency time sequence, and determines the fusion defect time and the defective data source.

[0139] The data matching and processing module 103 is used to match multiple homologous acquisition data and multiple heterologous simultaneous data from the multiple heterogeneous acquisition data according to the fusion defect time and the defective data source.

[0140] In an embodiment of the present invention, the data matching and processing module 103 screens multiple homologous acquisition data collected from the defective data source from the multiple frequency time sequence data, and matches multiple heterologous simultaneous data with the acquisition time being the fusion defect time from the multiple frequency time sequence data.

[0141] The defect prediction calculation module 104 is used to determine the level prediction time period, select multiple level time period data from the multiple homologous acquisition data for defect prediction, and calculate the target defect data.

[0142] In an embodiment of the present invention, the defect prediction calculation module 104 determines the level prediction time period, then divides the level prediction time period into two symmetric segments with the fusion defect time as the origin. The period before the origin is the pre-level prediction period, and the period after the origin is the post-level prediction period. Multiple level time period data with the acquisition time belonging to the pre-level prediction period or the post-level prediction period are selected from the multiple homologous acquisition data. Then, based on the multiple level time period data, defect prediction is performed on the data not collected by the defective data source at the fusion defect time, and the target defect data is calculated. Specifically, the calculation formula for the target defect data is:

[0143]

[0144] Where n represents the nth level prediction period, Bj is the target defect data for the nth-level prediction period, -n j is the period before grade prediction for the nth-level prediction period, n j is the period after grade prediction for the jth-level prediction period is the grade period data at time i of the jth-level prediction period is the period length at time i of the jth-level prediction period, and k is a preset period length decomposition factor

[0145] The comparison and verification processing module 105 is used to perform comparison and verification on the target defect data based on multiple pieces of the heterogeneous simultaneous data, and when the verification fails, gradually expand the grade prediction period until the verification passes

[0146] In an embodiment of the present invention, the comparison and verification processing module 105 performs a comparative analysis on the target defect data according to multiple pieces of heterogeneous simultaneous data, constructs a standard verification interval for the target defect data, and then, based on the standard verification interval, performs comparison and verification on the target defect data to determine whether the verification passes. Specifically, if the target defect data is within the standard verification interval, it is determined that the verification passes; if the target defect data is not within the standard verification interval, it is determined that the verification fails. At this time, a grade expansion instruction is generated, and according to the grade expansion instruction, the grade prediction period is gradually expanded, and with the expanded grade prediction period, multiple pieces of grade period data are reselected for defect prediction, and new target defect data is calculated

[0147] Further Figure 9 shows a structural block diagram of the comparison and verification processing module 105 in the system provided by an embodiment of the present invention

[0148] Specifically, in another preferred embodiment provided by the present invention, the comparison and verification processing module 105 specifically includes

[0149] The interval construction unit 1051 is used to analyze multiple pieces of the heterogeneous simultaneous data and construct a standard verification interval

[0150] The comparison and verification unit 1052 is used to perform comparison and verification on the target defect data according to the standard verification interval to determine whether the verification passes

[0151] The instruction generation unit 1053 is used to generate a grade expansion instruction when the verification fails

[0152] The gradual expansion unit 1054 is used to gradually expand the grade prediction period according to the grade expansion instruction

[0153] Further, the multi-source data fusion system for heterogeneous data sources further includes

[0154] The multi-source data fusion module 106 is used to perform multi-source data fusion processing.

[0155] In an embodiment of the present invention, after verification is passed, the multi-source data fusion module 106 determines valid defect data. Then, in multiple frequency-time series data, according to the time series of the valid defect data, the valid defect data is inserted and supplemented, and then data collation is performed to obtain a complete time series data set. Furthermore, multi-source data fusion is performed on the complete time series data set to generate multi-source fusion data.

[0156] It should be understood that although the steps in the flowcharts of the embodiments of the present invention are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0157] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0158] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention shall be subject to the appended claims.

Claims

1. A multi-source data fusion method for heterogeneous data sources, characterized in that: The method specifically comprises the following steps: receiving a data fusion request, determining a fusion period frequency and a plurality of heterogeneous data sources, and collecting a plurality of heterogeneous collected data from the plurality of heterogeneous data sources according to the fusion period frequency; Based on the fusion period frequency, defect identification is performed on the plurality of heterogeneous collected data to determine the fusion defect time and defect data source; According to the fusion defect time and the defect data source, matching multiple homogeneous collected data and multiple heterogeneous simultaneous data from the multiple heterogeneous collected data; Determine a level estimation period, select multiple level period data from the multiple homologous collected data for defect estimation, and calculate target defect data; Based on the multiple heterogeneous simultaneous data, the target defect data is verified by comparison, and when the verification fails, the level estimation period is gradually extended until the verification passes; Perform multi-source data fusion processing.

2. The multi-source data fusion method of heterogeneous data sources according to claim 1 is characterized in that: The receiving of the data fusion request, determining the fusion period frequency and the plurality of heterogeneous data sources, and collecting the plurality of heterogeneous collected data from the plurality of heterogeneous data sources according to the fusion period frequency specifically comprises the following steps: receiving a data fusion request; Identifying the data fusion request, determining a fusion period frequency and a plurality of heterogeneous data sources; Determining a data collection period according to the fusion period frequency; Creating a data collection task according to the data collection period and the plurality of heterogeneous data sources; According to the data collection task, a plurality of heterogeneous collection data are collected.

3. The multi-source data fusion method of heterogeneous data sources according to claim 1 is characterized in that: The method of performing defect identification on the plurality of heterogeneous collected data based on the fusion period frequency and determining the fusion defect time and defect data source specifically includes the following steps: Determining a fusion frequency sequence according to the fusion period frequency; According to the fusion frequency time series, a plurality of frequency time series data are screened from the plurality of heterogeneous collected data; Based on the fused frequency time series, defect identification is performed on a plurality of the frequency time series data to determine the fused defect time and defect data source.

4. The multi-source data fusion method of heterogeneous data sources according to claim 3 is characterized in that: The matching of the plurality of homogeneous collected data and the plurality of heterogeneous simultaneous data from the plurality of heterogeneous collected data according to the fusion defect time and the defect data source specifically comprises the following steps: According to the defect data source, a plurality of homologous collected data are screened from the plurality of frequency time series data; According to the fusion defect time, a plurality of heterogeneous simultaneous data are matched from a plurality of the frequency timing data.

5. The multi-source data fusion method of heterogeneous data sources according to claim 1 is characterized in that: Determining the level estimation period, selecting multiple level period data from the multiple homologous collected data for defect estimation, and calculating the target defect data specifically include the following steps: Determine the grade estimate period; Taking the fusion defect time as the origin, dividing the level estimation period into a period before level estimation and a period after level estimation; According to the period before the grade estimation and the period after the grade estimation, a plurality of grade period data are selected from the plurality of homologous collected data; Based on the multiple grade period data, defect prediction is performed and target defect data is calculated.

6. The multi-source data fusion method of heterogeneous data sources according to claim 5 is characterized in that: The calculation formula of the target defect data is: Among them, j represents the jth level estimation period, B j is the target defect data for the j-th level estimation period, -n j is the period before the level estimation of the j-th level estimation period, n j is the post-level estimation period of the j-th level estimation period, is the level period data of time i in the j-th level estimation period, is the time period length of time i in the j-th level estimation period, and k is the preset time period length decomposition factor.

7. The multi-source data fusion method of heterogeneous data sources according to claim 1 is characterized in that: The step of comparing and verifying the target defect data based on the multiple heterogeneous simultaneous data and gradually extending the level estimation period when the verification fails specifically includes the following steps: Analyze the multiple heterogeneous simultaneous data to construct a standard verification interval; According to the standard verification interval, the target defect data is verified against each other to determine whether the verification is passed; When the verification fails, a level extension instruction is generated; According to the level extension instruction, the level estimation time period is extended level by level.

8. The multi-source data fusion method of heterogeneous data sources according to claim 3 is characterized in that: The multi-source data fusion process specifically includes the following steps: When verification is passed, valid defect data is determined; Among the plurality of frequency time series data, the valid defect data is supplemented, and the data is sorted to generate a complete time series data set; Multi-source data fusion is performed on the complete time series data set to generate multi-source fused data.

9. A multi-source data fusion system for heterogeneous data sources, characterized in that: The system includes a multi-source data acquisition module, a data defect identification module, a data matching processing module, a defect estimation calculation module, a comparison verification processing module and a multi-source data fusion module, wherein: A multi-source data acquisition module is used to receive a data fusion request, determine a fusion period frequency and a plurality of heterogeneous data sources, and collect a plurality of heterogeneous data from the plurality of heterogeneous data sources according to the fusion period frequency; A data defect identification module, used to identify defects of the plurality of heterogeneous collected data based on the fusion period frequency, and determine the fusion defect time and defect data source; A data matching processing module, used for matching a plurality of homogeneous collected data and a plurality of heterogeneous simultaneous data from a plurality of heterogeneous collected data according to the fusion defect time and the defect data source; A defect estimation calculation module is used to determine a level estimation period, select multiple level period data from multiple homologous collected data for defect estimation, and calculate target defect data; A comparison and verification processing module, used for performing comparison and verification on the target defect data based on the multiple heterogeneous simultaneous data, and when the verification fails, gradually extending the level estimation period until the verification passes; The multi-source data fusion module is used to perform multi-source data fusion processing.

10. The multi-source data fusion system of heterogeneous data sources according to claim 9, characterized in that: The control verification processing module specifically includes: An interval construction unit, used for analyzing the plurality of heterogeneous simultaneous data to construct a standard verification interval; A comparison and verification unit, used to compare and verify the target defect data according to the standard verification interval to determine whether the verification is passed; An instruction generation unit, used for generating a level extension instruction when the verification fails; A step-by-step expansion unit is used to expand the level estimation time period step-by-step according to the level expansion instruction.

Images