Multi-source heterogeneous data integration method and system based on water price measurement and calculation

Abstract

The invention belongs to the technical field of data processing, and provides a multi-source heterogeneous data integration method and system based on water price calculation, and the method comprises the steps: carrying out the input aging analysis of a plurality of pieces of water price data in a similar database, and obtaining a local induction value; the input time difference of the water price parameters and the difference between the acquisition time difference and the input time difference are reflected through the local induction value, and the difference is compared with the data acquisition period to reflect the relative difference degree of the two groups of water price data in the time dimension under the background of the whole data acquisition period. According to the method, the difference degree between different water price data in the time dimension can be measured, the water price data with the small difference degree is integrated, local data integration is carried out, and it is guaranteed that the integrated data has high timeliness.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular to a multi-source heterogeneous data integration method and system based on water price calculation. Background Art

[0002] In the field of water price calculation, the processing of multi-source heterogeneous data faces many challenges. Water price data from different sources differ in format, parameter integrity, and collection and entry time. These differences make it difficult to directly integrate and utilize the data, and data confusion seriously affects the accuracy and efficiency of water price calculation.

[0003] In the prior art, the data collection and entry times are different, which makes it difficult to integrate and utilize the data, seriously affecting the accuracy and efficiency of water price calculation, making it impossible to effectively integrate various types of data to draw reliable conclusions when conducting water price analysis. Therefore, this application identifies the integrity of multi-source water price data to obtain complete data, performs feature analysis on the complete multi-source water price data to obtain similar feature values, reflecting the differences in the quantile length range between water price data and the overall degree of difference in the values ​​of specific water price parameters, thereby helping to identify data with similar features, and summarize data with similar features, thereby improving the subsequent efficiency of data aggregation, and performing time-effectiveness analysis on the entry of multiple water price data in similar databases. The local summary value is obtained by analysis, which reflects the difference in the input time of water price parameters and the difference in the time difference between collection and input. It is helpful to measure the degree of difference between different water price data in the time dimension, integrate the water price data with smaller differences, conduct local data aggregation, and ensure that the aggregated data has high timeliness, obtain the local aggregation time, and perform future local aggregation work on multi-source heterogeneous data according to the local aggregation time, which solves the problem of differences in the collection time of multi-source heterogeneous data. Local aggregation is performed through a unified time standard, which reduces data confusion caused by inconsistent data collection time, improves the accuracy and efficiency of data processing, and provides time for orderly processing and integration of data.

[0004] To this end, the present invention provides a multi-source heterogeneous data aggregation method and system based on water price calculation. Summary of the invention

[0005] In order to make up for the deficiencies of the prior art, at least one technical problem raised in the background technology is solved.

[0006] The technical solution adopted by the present invention to solve its technical problem is: In the first aspect, a multi-source heterogeneous data integration method based on water price calculation includes: Perform integrity identification on multi-source heterogeneous water price data, obtain data integrity signals, and enter the corresponding water price data into the data warehouse; Perform feature analysis on multiple heterogeneous water price data in the data warehouse to obtain the quantile range difference and quantile length difference, and perform quantitative processing to generate data similarity signals and obtain a similarity database; The input time efficiency analysis is performed on multiple water price data in similar databases to obtain the input time difference value and input frequency difference value, and then quantified to obtain the local induction signal; Based on the local induction signal, the local integration time is obtained, and local integration work is performed on future multi-source heterogeneous data.

[0007] As a further solution of the present invention: the process of entering the data warehouse is as follows: Select any water price data; The number of water price parameters in the water price data is obtained, and if the number of water price parameters is equal to the parameter number threshold, a data complete signal is generated; Extract the water price data that generates the complete data signal, obtain the number of parameters in the water price data, and enter the water price data with the same number of parameters into the data warehouse.

[0008] As a further solution of the present invention: the method of obtaining the percentile range difference is: Respectively obtain the front quantile length and the back quantile length corresponding to all water price parameters in the water price data; Extract the maximum and minimum values ​​of the front quantile length and make a difference to get the front quantile range value; Extract the maximum and minimum posterior quantile lengths and make a difference to obtain the posterior quantile range value; The difference between the front quantile range and the back quantile range is summed to calculate the unit quantile range value; Randomly select two heterogeneous water price data, and subtract the corresponding unit quantile range values, take the absolute value, and get the quantile range difference.

[0009] As a further solution of the present invention: the method for obtaining the quantile length difference is: Randomly select two water price data combinations as feature analysis groups; The water price parameters in the heterogeneous water price data are sorted according to the order of parameter entry time, and the sets and collection ; The water price parameters corresponding to the heterogeneous water price data in the feature analysis group are substituted into the Euclidean calculation formula to calculate the quantile length difference.

[0010] As a further solution of the present invention: the construction process of the similarity database is as follows: Sum the quantile range difference and the quantile length difference to get the similarity eigenvalue; If the similarity feature value is less than or equal to the similarity feature threshold, a data similarity signal is generated, and the heterogeneous water price data corresponding to the generated data similarity signal are summarized into a similarity database.

[0011] As a further solution of the present invention: the method for obtaining the input time difference value is: From the similar database, we randomly select two water price data sets to combine them to obtain the time-effectiveness analysis group, and extract the entry time nodes of the water price parameters in the water price data to obtain the sets and collection ; Substitute the input time nodes corresponding to the water price data in the time-effectiveness analysis group into the Euclidean calculation formula to obtain the input time difference value.

[0012] As a further solution of the present invention: the recording time difference value is obtained in the following manner: Sort the unit recording time differences corresponding to the water price data in the time-effectiveness analysis group according to the time series to obtain the set and collection , and combine the elements to obtain the recording analysis group, and substitute the corresponding unit recording time difference into the Euclidean calculation formula to calculate the recording time difference value.

[0013] As a further solution of the present invention: the generation process of the local induction signal is as follows: The recording time difference value is summed with the input time difference value, and the ratio is calculated with the duration corresponding to the data collection cycle to obtain the local summary value; If the local induction value is less than or equal to the local induction threshold, a local induction signal is generated.

[0014] As a further solution of the present invention: obtaining the local integration time, and performing local integration work on future multi-source heterogeneous data, the process is as follows: The time analysis group that generates the local summary signal is marked as the local summary group, and the corresponding unit recording time difference is extracted, and the summation and mean calculation are performed to obtain the unit data recording time difference; The time difference of the unit data recording corresponding to all the water price data in the local summary group is calculated and the mean is calculated to obtain the local summary time, which is substituted into the formula to calculate the future summary time point.

[0015] In the second aspect, a multi-source heterogeneous data integration system based on water price calculation includes: Complete identification module: performs integrity identification on multi-source heterogeneous water price data, obtains data integrity signals, and enters the corresponding water price data into the data warehouse; Similarity screening module: Perform feature analysis on multiple heterogeneous water price data in the data warehouse, obtain the quantile range difference and quantile length difference, and perform quantitative processing to generate data similarity signals and obtain a similarity database; Local induction module: analyze the input time efficiency of multiple water price data in similar databases, obtain the input time difference value and input frequency difference value, and perform quantitative processing to obtain local induction signals; Aggregation prediction module: Based on the local induction signal, obtain the local aggregation time and perform local aggregation on future multi-source heterogeneous data.

[0016] The beneficial effects of the present invention are as follows: The present invention identifies the integrity of multi-source water price data to obtain complete data, performs feature analysis on the complete multi-source water price data to obtain similar feature values, and reflects the differences in the quantile length range between the water price data and the overall degree of difference in the values ​​of specific water price parameters through the similar feature values, thereby facilitating the identification of data with similar features, and summarizing the data with similar features, thereby improving the efficiency of subsequent data aggregation; The present invention performs input time analysis on multiple water price data in similar databases to obtain local summary values, which reflect the input time difference of water price parameters and the difference between the collection and input time, and compares this difference with the data collection cycle to reflect the relative difference between the two groups of water price data in the time dimension under the background of the entire data collection cycle, which helps to measure the difference between different water price data in the time dimension, integrates water price data with smaller differences, performs local data aggregation, and ensures that the aggregated data has high timeliness; The present invention is based on local inductive signals, extracts the unit recording time difference corresponding to the water price data in the local integration group, and performs quantization processing to obtain the local integration time. Future local integration work is performed on multi-source heterogeneous data according to the local integration time, which solves the problem of differences in the collection time of multi-source heterogeneous data. Local integration is performed through a unified time standard, which reduces data confusion caused by inconsistent data collection time, improves the accuracy and efficiency of data processing, and provides time for orderly processing and integration of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below in conjunction with the accompanying drawings.

[0018] Figure 1 It is a flowchart of the steps of a multi-source heterogeneous data integration method based on water price calculation of the present invention; Figure 2 It is a flow chart of a multi-source heterogeneous data integration system based on water price calculation of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods. Example

[0020] like Figure 1 As shown, a multi-source heterogeneous data aggregation method based on water price calculation according to an embodiment of the present invention includes: Step 1: Perform integrity identification on multi-source heterogeneous water price data to obtain data integrity signals, and enter the water price data corresponding to the data integrity signals into the data warehouse; In some embodiments, one water price data is arbitrarily selected from the multi-source heterogeneous water price data; Get the number of water price parameters in the water price data and compare it with the parameter number threshold. The process is as follows: If the number of water price parameters is equal to the parameter number threshold, it means that the water price parameters in the analyzed water price data are complete, and a data completeness signal is generated; If the number of data parameters is not equal to the parameter number threshold, it means that the water price parameters in the analyzed water price data are incomplete, and a data incomplete signal is generated; It should be noted that the parameter data threshold is obtained by personnel in the technical field based on the number of parameters collected or entered historically; Extract the water price data that generates the complete data signal, obtain the number of parameters in the water price data, and enter the water price data with the same number of parameters into the data warehouse; Step 2: Perform feature analysis on multiple heterogeneous water price data in the data warehouse to obtain feature analysis information, wherein the feature analysis information includes quantile range difference and quantile length difference, quantile range difference and quantile length difference are quantified to obtain similar feature values, and compared with similar feature thresholds to generate data similarity signals and build a similarity database; In some embodiments, a water price data is randomly selected from the data warehouse; Extract the digits before and after the decimal point corresponding to all water price parameters in the water price data; Get the front quantile length and the back quantile length respectively; Extract the maximum and minimum values ​​of the front quantile length and make a difference to get the front quantile range value; Extract the maximum and minimum posterior quantile lengths and make a difference to obtain the posterior quantile range value; The difference between the front quantile range and the back quantile range is summed to calculate the unit quantile range value; Arbitrarily select two water price data and combine them to obtain multiple feature analysis groups; Arbitrarily select a set of feature analysis groups; Difference the unit quantile range values ​​corresponding to the water price data, take the absolute value, and obtain the quantile range difference; In the feature analysis group, the water price parameters in the water price data are sorted according to the order of parameter entry time, and the sets and collection ;; Where m represents the total number of water price parameters in the water price data, and the set It is represented as a set corresponding to one of the water price data in the feature analysis group. It is represented as a set corresponding to another water price data in the feature analysis group; Substitute the water price parameters corresponding to the water price data in the feature analysis group into the Euclidean calculation formula: , calculate the quantile length difference ,in, Represented as a set within a feature analysis group The mth water price parameter in Represented as a set within a feature analysis group The mth water price parameter in , where m represents the total number of water price parameters; The purpose of using the Euclidean calculation formula is to calculate the distance between two points in a multidimensional space. The water price parameter in each water price data is now regarded as the coordinate of a point in the multidimensional space. For the two water price data in the feature analysis group, their respective water price parameters form a set of coordinates of two points. Therefore, the size of the distance intuitively reflects the difference in the water price parameter values ​​of the two heterogeneous water price data, which helps to aggregate data with similar characteristics and facilitate the subsequent data aggregation. Sum the quantile range difference and the quantile length difference to get the similarity eigenvalue; It can be understood that the meaning of similar feature values ​​is: used to judge whether two sets of water price data are similar. Specifically, on the one hand, the quantile range difference reflects the difference in quantile length range between water price data, and on the other hand, the quantile length difference reflects the overall difference between the two sets of data in the specific water price parameter values. From the perspective of quantitative calculation, it reflects the characteristic differences between water price data from different dimensions, which helps to identify data with similar characteristics, summarize data with similar characteristics, and improve the efficiency of subsequent data aggregation; The similarity feature value is compared with the similarity feature threshold, and the process is as follows: If the similarity feature value is greater than the similarity feature threshold, it means that the difference in the quantile length range and the specific water price parameter value is large, generating a data difference signal; If the similarity feature value is less than or equal to the similarity feature threshold, it means that the difference in the quantile length range and the specific water price parameter value is small, and a data similarity signal is generated, and the heterogeneous water price data corresponding to the generated data similarity signal are summarized as a similar database; The specific implementation scheme of the embodiment of the present invention is: to identify the integrity of multi-source water price data to obtain complete data, to perform feature analysis on the complete multi-source water price data to obtain similar feature values, and to reflect the differences in the quantile length range between the water price data and the overall degree of difference in the values ​​of specific water price parameters through similar feature values, thereby helping to identify data with similar features, and to summarize data with similar features, thereby improving the efficiency of subsequent data aggregation. Example

[0021] like Figure 1 As shown, based on Example 1, a multi-source heterogeneous data aggregation method based on water price calculation according to an embodiment of the present invention includes: Step 3: Perform input timeliness analysis on multiple water price data in similar databases to obtain timeliness analysis data, wherein the timeliness analysis data includes input time difference value and sampling time difference value, quantify the input time difference value and the input frequency difference value to obtain a local summary value, and analyze it with the local summary threshold to obtain a local summary signal; In some embodiments, two water price data are randomly selected from a similar database and combined to obtain multiple time-effect analysis groups; In the time-effectiveness analysis group, the entry time nodes of water price parameters in the water price data are extracted to obtain the sets and collection ; Among them, m represents the total number of time nodes corresponding to the water price parameters in the water price data, and the set It is represented as a set corresponding to one of the water price data in the time-effectiveness analysis group. It is represented as a set corresponding to another water price data in the time-effectiveness analysis group; It should be noted that the total number of water price parameters in the water price data is equal to the total number of time nodes corresponding to the water price parameters in the water price data; Substitute the input time nodes corresponding to the water price data in the time-effectiveness analysis group into the Euclidean calculation formula: , calculate the input time difference ,in, Represented as a set within the time-effect analysis group The input time node corresponding to the mth water price parameter in, Represented as a set within the time-effect analysis group The entry time node corresponding to the mth water price parameter in , where m represents the total number of entry time nodes corresponding to the water price parameter; In the time-effect analysis group, the collection time nodes of the water price parameters in the water price data are extracted, and the time nodes are subtracted from the corresponding input time nodes to obtain the unit collection and recording time difference; Sort the unit recording time differences corresponding to the water price data in the time-effectiveness analysis group according to the time series to obtain the set and collection ; in, It is represented by the total number of time nodes corresponding to the water price parameters in the water price data, the set It is represented as a set of unit recording time differences corresponding to one of the water price data in the time analysis group. It represents the time difference set of the unit recording corresponding to another water price data in the time-effectiveness analysis group; Will gather Elements and collections in Elements are combined to obtain multiple recording and analysis groups; For example, and A recording and analysis group will and A recording and analysis group will and A recording and analysis group will and A recording and analysis team; Substitute the recording time difference of the unit corresponding to the recording analysis group into the Euclidean calculation formula: , calculate the recording time difference ,in, Expressed as Collection within the analysis group The corresponding unit recording time difference, Expressed as Collection within the analysis group The corresponding unit recording time difference, m-1 represents the total number of time-effect analysis groups; The time difference will be recorded The local summary value is obtained by summing the input time difference value and calculating the ratio with the duration corresponding to the data collection period; It can be understood that the meaning of the local summary value is: reflecting the difference in the input time of the water price parameters and the difference in the time difference between the collection and input, and comparing this difference with the data collection cycle to reflect the relative difference between the two sets of water price data in the time dimension in the context of the entire data collection cycle, which helps to measure the difference between different water price data in the time dimension, and integrate the water price data with smaller differences, conduct local data aggregation, and ensure that the aggregated data has a high timeliness; The purpose of using the Euclidean calculation formula is that the Euclidean calculation formula is a common distance measurement method, which can quantify the differences in the entry time nodes of water price parameters in water price data and the differences in the unit recording time difference, and reasonably measure the degree of difference between different time nodes, thereby measuring the similarity of different water price data in the time dimension; The local induction value is compared with the local induction threshold as follows: If the local induction value is greater than the local induction threshold, it means that the water price data has a high similarity in the time dimension, generating a recombination signal; If the local induction value is less than or equal to the local induction threshold, it means that the water price data has a low similarity in the time dimension, and a local induction signal is generated; The specific implementation scheme of the embodiment of the present invention is: perform input time analysis on multiple water price data in similar databases to obtain local summary values, and reflect the input time differences of water price parameters and the differences in collection and input time differences through local summary values, and compare this difference with the data collection cycle to reflect the relative difference between the two groups of water price data in the time dimension in the context of the entire data collection cycle, which helps to measure the difference between different water price data in the time dimension, and integrate water price data with smaller differences, perform local data aggregation, and ensure that the aggregated data has high timeliness. Example

[0022] like Figure 1 As shown, based on Example 1 and Example 2, a multi-source heterogeneous data aggregation method based on water price calculation according to an embodiment of the present invention includes: Step 4: Based on the local induction signal, obtain the local integration time, and perform local integration work on future multi-source heterogeneous data according to the local integration time; In some embodiments, the temporal analysis group that generates the local summary signal is marked as a local summary group; Extract the unit recording time difference corresponding to the water price data in the local summary group, perform summation and mean calculation, and obtain the unit data recording time difference; The time difference of recording unit data corresponding to all water price data in the local aggregation group is calculated by summing up the mean to obtain the local aggregation time; Substituting the local integration time into the formula: , calculate the future summary time point, where is the current aggregation time point for the water price data in the local aggregation group, is the local integration time; The specific implementation plan of the embodiment of the present invention is: based on the local inductive signal, the unit recording time difference corresponding to the water price data in the local integration group is extracted, and quantified to obtain the local integration time, and the local integration work of multi-source heterogeneous data is predicted in the future according to the local integration time, which solves the problem of differences in the collection time of multi-source heterogeneous data (water price data). Local integration is performed through a unified time standard, which reduces data confusion caused by inconsistent data collection time, improves the accuracy and efficiency of data processing, and provides time for orderly processing and integration of data. Example

[0023] like Figure 2 As shown, a multi-source heterogeneous data integration system based on water price calculation according to an embodiment of the present invention includes the following modules: Complete identification module: performs integrity identification on multi-source heterogeneous water price data, obtains data integrity signals, and enters the corresponding water price data into the data warehouse; Similarity screening module: Perform feature analysis on multiple heterogeneous water price data in the data warehouse, obtain the quantile range difference and quantile length difference, and perform quantitative processing to generate data similarity signals and obtain a similarity database; Local induction module: analyze the input time efficiency of multiple water price data in similar databases, obtain the input time difference value and input frequency difference value, and perform quantitative processing to obtain local induction signals; Aggregation prediction module: Based on the local induction signal, obtain the local aggregation time and perform local aggregation of future multi-source heterogeneous data.

[0024] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-source heterogeneous data integration method based on water price calculation, characterized by: include: Perform integrity identification on multi-source heterogeneous water price data, obtain data integrity signals, and enter the corresponding water price data into the data warehouse; Perform feature analysis on multiple heterogeneous water price data in the data warehouse to obtain the quantile range difference and quantile length difference, and perform quantitative processing to generate data similarity signals and obtain a similarity database; The input time efficiency analysis is performed on multiple water price data in similar databases to obtain the input time difference value and input frequency difference value, and then quantified to obtain the local induction signal; Based on the local induction signal, the local integration time is obtained, and local integration work is performed on future multi-source heterogeneous data.

2. According to claim 1, a multi-source heterogeneous data integration method based on water price calculation is characterized by: The process of entering the data warehouse is as follows: Select any water price data; The number of water price parameters in the water price data is obtained, and if the number of water price parameters is equal to the parameter number threshold, a data complete signal is generated; Extract the water price data that generates the complete data signal, obtain the number of parameters in the water price data, and enter the water price data with the same number of parameters into the data warehouse.

3. The multi-source heterogeneous data integration method based on water price calculation according to claim 2 is characterized by: The way to obtain the percentile range difference is: Respectively obtain the front quantile length and the back quantile length corresponding to all water price parameters in the water price data; Extract the maximum and minimum values ​​of the front quantile length and make a difference to get the front quantile range value; Extract the maximum and minimum posterior quantile lengths and make a difference to obtain the posterior quantile range value; The difference between the front quantile range and the back quantile range is summed to calculate the unit quantile range value; Randomly select two heterogeneous water price data, and subtract the corresponding unit quantile range values, take the absolute value, and get the quantile range difference.

4. The multi-source heterogeneous data integration method based on water price calculation according to claim 2 is characterized by: The way to obtain the quantile length difference is: Randomly select two water price data combinations as feature analysis groups; The water price parameters in the heterogeneous water price data are sorted according to the order of parameter entry time, and the sets and collection ; The water price parameters corresponding to the heterogeneous water price data in the feature analysis group are substituted into the Euclidean calculation formula to calculate the quantile length difference.

5. The multi-source heterogeneous data integration method based on water price calculation according to claim 1 is characterized by: The construction process of the similarity database is as follows: Sum the quantile range difference and the quantile length difference to get the similarity eigenvalue; If the similarity feature value is less than or equal to the similarity feature threshold, a data similarity signal is generated, and the heterogeneous water price data corresponding to the generated data similarity signal are summarized into a similarity database.

6. The multi-source heterogeneous data integration method based on water price calculation according to claim 5 is characterized by: The method for obtaining the input time difference value is: From the similar database, we randomly select two water price data sets to combine them to obtain the time-effectiveness analysis group, and extract the entry time nodes of the water price parameters in the water price data to obtain the sets and collection ; Substitute the input time nodes corresponding to the water price data in the time-effectiveness analysis group into the Euclidean calculation formula to obtain the input time difference value.

7. The multi-source heterogeneous data integration method based on water price calculation according to claim 6 is characterized by: The method for obtaining the recording time difference is as follows: Sort the unit recording time differences corresponding to the water price data in the time-effectiveness analysis group according to the time series to obtain the set and collection , and combine the elements to obtain the recording analysis group, and substitute the corresponding unit recording time difference into the Euclidean calculation formula to calculate the recording time difference value.

8. The multi-source heterogeneous data integration method based on water price calculation according to claim 1 is characterized by: The generation process of the local induction signal is as follows: The recording time difference value is summed with the input time difference value, and the ratio is calculated with the duration corresponding to the data collection cycle to obtain the local summary value; If the local induction value is less than or equal to the local induction threshold, a local induction signal is generated.

9. The multi-source heterogeneous data integration method based on water price calculation according to claim 1 is characterized by: Get the local integration time and perform local integration of future multi-source heterogeneous data. The process is as follows: The time analysis group that generates the local summary signal is marked as the local summary group, and the corresponding unit recording time difference is extracted, and the summation and mean calculation are performed to obtain the unit data recording time difference; The time difference of the unit data recording corresponding to all the water price data in the local summary group is calculated and the mean is calculated to obtain the local summary time, which is substituted into the formula to calculate the future summary time point.

10. A multi-source heterogeneous data integration system based on water price calculation, characterized by: include: Complete identification module: performs integrity identification on multi-source heterogeneous water price data, obtains data integrity signals, and enters the corresponding water price data into the data warehouse; Similarity screening module: Perform feature analysis on multiple heterogeneous water price data in the data warehouse, obtain the quantile range difference and quantile length difference, and perform quantitative processing to generate data similarity signals and obtain a similarity database; Local induction module: analyze the input time efficiency of multiple water price data in similar databases, obtain the input time difference value and input frequency difference value, and perform quantitative processing to obtain local induction signals; Aggregation prediction module: Based on the local induction signal, obtain the local aggregation time and perform local aggregation on future multi-source heterogeneous data.

Images