A multi-source heterogeneous data integration storage method for traffic engineering construction

Abstract

The present invention belongs to the technical field of data storage, and provides a multi-source heterogeneous data integrated storage method for traffic engineering construction. The method performs standardized processing, feature extraction and cluster analysis on the multi-source heterogeneous data in the process of traffic engineering construction, and based on the clustering results, performs membership analysis on the standardized data to obtain feature clustering clusters and main feature clustering clusters for each standardized data; for historical multi-source heterogeneous data, data types are divided according to the main feature clustering clusters; for new multi-source heterogeneous data acquired in real time, the distribution characteristics of the standardized data in the feature clustering clusters are analyzed, and the revised membership between the new multi-source heterogeneous data and each feature clustering cluster center is obtained in combination with the membership; the new multi-source heterogeneous data is divided into data types according to the revised membership. The present invention ensures the accuracy of data classification by correcting the membership, thereby ensuring the accuracy and efficiency of data storage.

Description

Technical Field

[0001] The present invention relates to the field of data protection technology, and in particular to a multi-source heterogeneous data integration storage method for traffic engineering construction. Background Art

[0002] During the construction of transportation projects, the data generated often comes from a variety of different sensors and data sources, such as construction monitoring data of infrastructure such as bridges, roads, and tunnels, which contains real-time information such as temperature, humidity, stress strain, and vibration. In addition, there are historical data such as meteorological data, construction progress, and traffic flow. These data are not only heterogeneous and in various formats (such as text, numerical values, images, and videos), but also have different time granularities, including both real-time data and long-term accumulated historical data. This multi-source heterogeneous data characteristic brings challenges to the integrated storage of data.

[0003] Traditional data storage methods cannot effectively handle the heterogeneous data generated during the construction of transportation projects, resulting in inefficient data management and low data utilization. In addition, it is difficult to provide efficient data support in subsequent analysis and applications.

[0004] Therefore, in order to ensure the effective management and utilization of these data, there is an urgent need for a method that can effectively standardize multi-source heterogeneous data, and when processing real-time data, quickly and accurately classify it to the corresponding storage nodes and ensure the accuracy and efficiency of data storage. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multi-source heterogeneous data integration storage method for traffic engineering construction.

[0006] According to the present invention, a multi-source heterogeneous data integration storage method for traffic engineering construction is provided, the method comprising:

[0007] Collecting multi-source heterogeneous data during the construction of a traffic project, and performing standardization processing on the multi-source heterogeneous data to obtain standardized data;

[0008] Performing feature extraction on the standardized data to obtain a feature vector for each piece of the standardized data;

[0009] According to the feature vector, cluster analysis is performed on the standardized data, and based on the clustering result, membership analysis is performed on the standardized data to obtain a feature clustering cluster and a main feature clustering cluster of each piece of the standardized data;

[0010] For historical multi-source heterogeneous data, data types are divided according to the main feature clustering clusters;

[0011] For the new multi-source heterogeneous data acquired in real time, analyzing the distribution characteristics of the standardized data in the feature clustering cluster, and combining the membership degree to obtain the modified membership degree between the new multi-source heterogeneous data and each of the feature clustering cluster centers;

[0012] The new multi-source heterogeneous data are divided into data types according to the modified membership degree.

[0013] In some embodiments of the present invention, the multi-source heterogeneous data are subjected to standardization processing, and before that, the step further includes: performing data cleaning on the multi-source heterogeneous data.

[0014] In some embodiments of the present invention, the multi-source heterogeneous data are standardized, including: converting all the multi-source heterogeneous data into decimal data to obtain standardized data.

[0015] In some embodiments of the present invention, feature extraction is performed on the standardized data to obtain a feature vector of each piece of the standardized data, including:

[0016] Extracting data features of each piece of the standardized data, wherein the data features include the number of peaks, the maximum peak, and the time difference between two adjacent peaks;

[0017] The data features are integrated into a feature vector as a feature vector of each piece of the standardized data.

[0018] In some embodiments of the present invention, cluster analysis is performed on the standardized data according to the feature vector, including:

[0019] According to the feature vector, the elbow method is used to determine the K value for K-means clustering of each of the standardized data;

[0020] According to the K value, the standardized data is clustered by a fuzzy C-means clustering algorithm to obtain K clusters for each piece of the standardized data.

[0021] In some embodiments of the present invention, based on the clustering results, the standardized data is subjected to membership analysis to obtain a feature clustering cluster and a main feature clustering cluster for each piece of the standardized data, including:

[0022] Obtaining the Euclidean distance from the standardized data to the center of each cluster;

[0023] Analyzing the distance ratio relationship between the Euclidean distance from the standardized data to the cluster center and the sum of the Euclidean distances to all the cluster centers to obtain the membership degree between the standardized data and the cluster center;

[0024] Setting a membership threshold, recording the cluster whose membership corresponding to the standardized data is greater than the cluster corresponding to the membership threshold as a characteristic cluster of the standardized data;

[0025] The cluster corresponding to the maximum membership degree corresponding to the standardized data is recorded as the main feature cluster of the standardized data.

[0026] In some embodiments of the present invention, for historical multi-source heterogeneous data, data types are divided according to the main feature clustering clusters, including:

[0027] For historical multi-source heterogeneous data, data with the same main feature cluster corresponding to the standardized data are divided into one data type.

[0028] In some embodiments of the present invention, for the new multi-source heterogeneous data acquired in real time, the distribution characteristics of the standardized data in the feature clustering cluster are analyzed, and the membership degree is combined to obtain the modified membership degree between the new multi-source heterogeneous data and each of the feature clustering cluster centers, including:

[0029] For the new multi-source heterogeneous data acquired in real time, standardization processing and feature extraction are first performed, and then membership analysis is performed to obtain the original membership and original feature clustering clusters corresponding to each new standardized data in the new multi-source heterogeneous data;

[0030] In combination with the original membership, analyzing the proportional relationship between the amount of historical standardized data in the feature clustering cluster and the total amount of historical standardized data in all original feature clustering clusters corresponding to the new standardized data, to obtain the distribution characteristics of the standardized data in the feature clustering cluster;

[0031] The distribution features are used as weights to perform weighted processing on the original memberships, thereby obtaining modified memberships between the new standardized data in the new multi-source heterogeneous data and each of the feature cluster centers.

[0032] In some embodiments of the present invention, the original membership is combined to analyze the proportional relationship between the amount of historical standardized data in the feature clustering cluster and the total amount of historical standardized data in all original feature clustering clusters corresponding to the new standardized data, and the distribution characteristics of the standardized data in the feature clustering cluster are obtained, including:

[0033] Obtaining the number of historical standardized data in the feature clustering cluster;

[0034] Get the total number of historical standardized data in all original feature clusters corresponding to the new standardized data;

[0035] Using the original membership as a weight, processing the total quantity to obtain a weighted total quantity;

[0036] The proportional relationship between the quantity and the weighted total quantity is calculated to obtain the distribution characteristics of the standardized data in the feature clustering cluster.

[0037] In some embodiments of the present invention, dividing the new multi-source heterogeneous data into data types according to the modified membership degree includes:

[0038] Setting a membership threshold, recording the revised membership corresponding to the new standardized data greater than the original feature cluster corresponding to the membership threshold as the revised feature cluster of the new standardized data;

[0039] The original feature cluster corresponding to the maximum modified membership degree corresponding to the new standardized data is recorded as the modified main feature cluster of the new standardized data;

[0040] The newly standardized data is divided into the data type to which the corresponding modified main feature cluster belongs.

[0041] It can be seen from the above embodiments that the multi-source heterogeneous data integration storage method for traffic engineering construction provided by the embodiments of the present invention has the following beneficial effects:

[0042] The present invention realizes the classification and storage of multi-source heterogeneous data by extracting features, clustering analysis and membership calculation of multi-source heterogeneous data in the process of traffic engineering construction. First, feature extraction is performed on the standardized multi-source heterogeneous data, and the feature vector in the standardized data is extracted by a sliding window or other methods to prepare for the subsequent clustering analysis. Based on the extracted feature vector, the standardized data is classified using a clustering algorithm. First, the number of clusters K is determined by the K-means algorithm, and then the standardized data is clustered using the fuzzy C-means clustering algorithm to calculate the membership between each standardized data and the cluster center. According to the membership, the main feature clustering cluster and the secondary feature clustering cluster of each standardized data are determined. For historical multi-source heterogeneous data, the data with the same main feature clustering cluster corresponding to the standardized data are divided into one data type. For the new multi-source heterogeneous data acquired in real time, the single-category data (only the main feature clustering cluster) is directly stored; while the remaining multi-category data (there are multiple feature clustering clusters) are corrected by weighted membership to ensure the accuracy of data classification. Through the above clustering analysis and membership calculation, the category of each data is finally determined and divided into corresponding storage nodes. Real-time multi-source heterogeneous data is dynamically allocated according to the clustering results, and historical multi-source heterogeneous data is stored according to the existing classification. The accuracy of data classification is ensured through the calculation and correction of membership, thus laying the foundation for subsequent efficient storage.

[0043] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A basic flow chart of a multi-source heterogeneous data integration storage method for traffic engineering construction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a multi-source heterogeneous data integration storage method for traffic engineering construction proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. Terms such as "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such article or device. In the absence of further restrictions, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the article or device including the element.

[0048] The specific scenario targeted by the present invention is the integrated management and storage of multi-source heterogeneous data during the construction of traffic engineering. In the scene of traffic engineering construction, there are usually a large number of sensors and monitoring equipment to collect data from different sources, such as construction monitoring data of infrastructure such as bridges, roads, and tunnels, including real-time information such as temperature, humidity, stress strain, and vibration. In addition, there are historical data such as meteorological data, construction progress, and traffic flow. These data are not only heterogeneous and in various formats (such as text, numerical values, images, videos, etc.), but also have different time granularity, including both real-time data and long-term accumulated historical data. In order to ensure the effective management and utilization of these data, the system needs to have the ability to standardize, classify, and compress storage, especially in terms of rapid acquisition of real-time data, reasonable allocation of storage nodes, and dynamic adjustment and optimized storage of historical data, so as to cope with the long-term and complex data processing needs of engineering projects.

[0049] The purpose of the present invention is to solve the efficiency problem of multi-source heterogeneous data storage and management during the construction of transportation projects, especially the complexity of historical and real-time data from different sensors and data sources in terms of format, timing, etc.

[0050] The following will introduce in detail a multi-source heterogeneous data integration storage method for traffic engineering construction provided by this embodiment in conjunction with the accompanying drawings.

[0051] See also Figure 1 , which shows the basic process of a multi-source heterogeneous data integration storage method for traffic engineering construction provided by an embodiment of the present invention.

[0052] like Figure 1 As shown, an embodiment of the present invention provides a multi-source heterogeneous data integration and storage method for traffic engineering construction, which specifically includes the following steps:

[0053] S100: Collect multi-source heterogeneous data during the construction of a transportation project, and perform standardization processing on the multi-source heterogeneous data to obtain standardized data.

[0054] Different types of time series data are collected during the construction of traffic engineering projects to obtain multi-source heterogeneous data, that is, various types of historical data and real-time data obtained through different sensors.

[0055] During the data collection process, the sensor may be disturbed by the external environment, resulting in missing or inaccurate data, such as sensor failure or noise interference. Therefore, in some embodiments of the present invention, data cleaning is performed on the acquired multi-source heterogeneous data to exclude data that cannot be used or has missing data.

[0056] In addition, since the sensor data in the process of traffic engineering construction may exist in multiple formats, some may be text data, some may be numerical data (may be in different bases), and the data formats of images, videos, etc. are also different, and data in different bases may cause errors or processing difficulties during calculation, it is necessary to standardize the multi-source heterogeneous data, that is, convert all the multi-source heterogeneous data into decimal data to obtain standardized data. Among them, the size and length of standardized data are different.

[0057] S200: Extract features from the standardized data to obtain a feature vector for each piece of standardized data.

[0058] Although the length of the standardized data obtained after standardization is different, it still has some characteristics of specific dimensions. Therefore, it is necessary to extract the characteristics of the standardized data to ensure that the standardized data can be clustered and analyzed through these characteristics, and some similar multi-source heterogeneous data can be summarized and classified to facilitate their subsequent integrated storage.

[0059] Therefore, feature extraction is performed on the standardized data to obtain a feature vector of each standardized data. Further steps include: extracting data features of each standardized data, the data features including the number of peaks, the maximum peak, and the time difference between two adjacent peaks; integrating the data features into a feature vector as the feature vector of each standardized data.

[0060] To further elaborate, the data obtained by different sensors are often expressed as time series, in which the fluctuation characteristics (such as peak values) can reflect the key changes in the operation status of the project. For example, in the stress data of a bridge, the peak value may mean an instantaneous change in the load. Therefore, each standardized data in the historical multi-source heterogeneous data can be traversed through a sliding window to obtain the peak value in each standardized data, as well as the value of each peak value and its corresponding position. By analyzing each standardized data, the fluctuation of each standardized data is represented by an extreme point.

[0061] After obtaining the peak values ​​of the standardized data, the peak position, maximum value, time difference between adjacent peak values, etc., cannot fully describe the changing trend of the data when they exist alone. Therefore, they must be combined to obtain a representative data feature. Therefore, the number of peak values, maximum peak value, and time difference between two adjacent peak values ​​(or the difference between data points) of each standardized data are integrated into a feature vector as the feature vector of the standardized data.

[0062] S300: performing cluster analysis on the standardized data according to the feature vectors, and performing membership analysis on the standardized data based on the clustering results to obtain a feature clustering cluster and a main feature clustering cluster for each piece of standardized data.

[0063] After obtaining the feature vector of each standardized data, it is necessary to classify all the standardized data and group the data of the same data type together, so it is necessary to cluster the standardized data. Therefore, firstly, cluster analysis is performed on the standardized data according to the feature vector to obtain the clustering result. Specifically, according to the feature vector, the elbow method is used to determine the K value for K-means clustering of each standardized data; according to the K value, the standardized data is clustered by the fuzzy C-means clustering algorithm to obtain K clusters for each standardized data.

[0064] Then, based on the clustering results, the standardized data is subjected to membership analysis to obtain the characteristic clustering cluster and the main characteristic clustering cluster of each standardized data. Specifically, the Euclidean distance from the standardized data to the center of each clustering cluster is obtained; the distance ratio relationship between the Euclidean distance from the standardized data to the center of the clustering cluster and the sum of the Euclidean distances to the centers of all clustering clusters is analyzed to obtain the membership between the standardized data and the centers of the clustering clusters; a membership threshold is set, and the clustering cluster corresponding to the standardized data with a membership greater than the membership threshold is recorded as the characteristic clustering cluster of the standardized data; the clustering cluster corresponding to the maximum membership of the standardized data is recorded as the main characteristic clustering cluster of the standardized data.

[0065] The calculation formula for the membership between the standardized data and the cluster center is:

[0066]

[0067] In the formula, Indicates The standardized data and The degree of membership between cluster centers; Indicates the number of clusters; Indicates Standardized data to The Euclidean distance between the cluster centers; Indicates Standardized data to The Euclidean distance between the cluster centers; Represents the fuzzy index, which is set to 1.5 to control the fuzziness of the membership. The larger the value, the smoother the membership.

[0068] Indicates Standardized data to and The distance ratio between the Euclidean distances of the centers of two clusters is larger. Standardized data to The larger the Euclidean distance between the cluster centers, the The standardized data and The farther the distance between the cluster centers, the Standardize data to The smaller the degree of membership between cluster centers; The exponent used to adjust the distance ratio affects the attribution of data points to each cluster.

[0069] The membership threshold is set to 90%, and the clusters with membership greater than 90% are recorded as the characteristic clusters of the standardized data. The clusters corresponding to the maximum membership are recorded as the main characteristic clusters of the standardized data, because in many cases, although the data may vaguely belong to multiple categories, there is always a main category that is most relevant to it.

[0070] S400: For historical multi-source heterogeneous data, data types are divided according to main feature clustering clusters.

[0071] For historical multi-source heterogeneous data, data with the same main feature clustering cluster corresponding to the standardized data are divided into one data type.

[0072] S500: for the new multi-source heterogeneous data acquired in real time, the distribution characteristics of the standardized data in the feature clustering cluster are analyzed, and the modified membership between the new multi-source heterogeneous data and each feature clustering cluster center is obtained by combining the membership.

[0073] After the historical multi-source heterogeneous data is classified and stored, the real-time multi-source heterogeneous data needs to be judged in real time and divided into corresponding data storage nodes. Before data division, the real-time multi-source heterogeneous data needs to be classified, and the data with only the main feature clustering cluster and no other feature clustering clusters is recorded as single-category data, which can be directly stored; but the rest of the data with other feature clustering clusters except the main feature clustering cluster is recorded as multi-category data. When analyzing, there may be two main feature clustering clusters with close membership, but due to errors or other interference, there will be errors in the main feature clustering clusters of the multi-source heterogeneous data. At this time, it needs to be corrected to reduce the impact of the error.

[0074] Based on the above analysis, in an embodiment of the present invention, for the new multi-source heterogeneous data acquired in real time, the distribution characteristics of the standardized data in the feature clustering cluster are analyzed, and combined with the membership degree, the modified membership degree between the new multi-source heterogeneous data and each feature clustering cluster center is obtained.

[0075] Further including: for the new multi-source heterogeneous data acquired in real time, firstly adopt the same method as described above in the embodiment of the present invention to perform standardization processing and feature extraction, and obtain the feature vector of each new standardized data in the new multi-source heterogeneous data; then perform membership analysis, that is, calculate the original membership between each new standardized data and each clustering cluster based on the clustering clusters obtained by clustering the historical multi-source heterogeneous data and the feature vector of each new standardized data, and then obtain several original feature clustering clusters and one original main feature clustering cluster for each new standardized data in the new multi-source heterogeneous data. It should be noted that the clustering clusters here are clustering clusters obtained through historical multi-source heterogeneous data, and unless otherwise specified in this article, the clustering clusters involved are all clustering clusters obtained through historical multi-source heterogeneous data.

[0076] When all the original feature clustering clusters corresponding to a new standardized data in the new multi-source heterogeneous data acquired in real time only contain the original main feature clustering cluster, it is determined to be single-category data and is divided into the storage node corresponding to the original feature clustering cluster; because the classification of these data is clear, they can be directly stored in the corresponding storage node, which can simplify the storage process and ensure that the new multi-source heterogeneous data is organized and stored according to clear categories, which is convenient for subsequent rapid retrieval and use.

[0077] When all the original feature clusters corresponding to a new standardized data in the new multi-source heterogeneous data acquired in real time are not only the original main feature clusters, it is determined to be multi-category data. Normally, the new standardized data can be stored in the storage node corresponding to its original main feature cluster, but due to the influence of various interference factors when the new multi-source heterogeneous data is acquired, sometimes its original main feature cluster does not represent that its data type is the data type corresponding to the original main feature cluster, and a small range of deviations will occur. Therefore, it is necessary to recalculate the membership between each cluster center of the multi-category data.

[0078] In an embodiment of the present invention, by analyzing the distribution characteristics of the standardized data in the feature clustering cluster, the original membership is weighted to obtain the revised membership between all the new standardized data in the new multi-source heterogeneous data and each feature clustering cluster center. Further, the number of historical standardized data in the feature clustering cluster is obtained, and then combined with the original membership, the proportional relationship between the number of historical standardized data in the feature clustering cluster and the total number of historical standardized data in all original feature clustering clusters corresponding to the new standardized data is analyzed to obtain the distribution characteristics of the standardized data in the feature clustering cluster. Then, the distribution characteristics are used as weights to perform weighted processing on the membership to obtain the revised membership between the new standardized data in the new multi-source heterogeneous data and each feature clustering cluster center.

[0079] Furthermore, in order to ensure that the correction process reflects the distribution of clusters and retains the intrinsic relationship between the new multi-source heterogeneous data and these clusters, so as to avoid excessive deviation caused by relying solely on the distribution of historical multi-source heterogeneous data, the participation of the original membership is retained in the revised membership. In this way, it can also be avoided that if the amount of historical data of a cluster is particularly large, then no matter how similar the new standardized data is to the cluster, it will be forcibly classified into the cluster, resulting in misclassification; on the contrary, if the amount of historical data of a cluster is small, but the new standardized data is actually very close to the center of this cluster, then the new standardized data may be mistakenly assigned to other larger clusters, although it essentially belongs to this small cluster. In some embodiments of the present invention, the original membership is combined to analyze the proportional relationship between the number of historical standardized data in the feature clustering cluster and the total number of historical standardized data in all original feature clustering clusters corresponding to the new standardized data, and obtain the distribution characteristics of the standardized data in the feature clustering cluster, specifically including: obtaining the number of historical standardized data in the feature clustering cluster; obtaining the total number of historical standardized data in all original feature clustering clusters corresponding to the new standardized data; using the original membership as a weight to process the total number to obtain a weighted total number; calculating the proportional relationship between the number and the weighted total number to obtain the distribution characteristics of the standardized data in the feature clustering cluster.

[0080] The calculation formula for the modified membership between the new standardized data and each feature cluster center in constructing new multi-source heterogeneous data is:

[0081]

[0082] In the formula, Indicates New standardized data and The modified membership between clusters; To indicate the New standardized data and The original membership between clusters; Indicates The number of historical standardized data in the clusters; Indicates The number of clusters contained in the new standardized data, that is, The number of original feature clusters of the new standardized data; Indicates New standardized data and The original membership between the original feature clusters; For the New standardized data The number of historical standardized data in the original feature clusters.

[0083] Will As the weighting coefficient, for the The number of historical standardized data in the clusters The larger the value, the more historical standardized data the cluster contains, indicating that the data type corresponding to the cluster is a more important and common data type. Therefore, the greater the possibility that the new standardized data belongs to this data type. New standardized data and The larger the modified membership between clusters, the larger the , represents the weighted result of the total number of historical standardized data in all original feature clusters corresponding to the new standardized data. The smaller the value, the smaller the number of historical standardized data in the original feature clusters corresponding to the new standardized data, and the New standardized data and The smaller the original membership between the original feature clusters, the smaller the data type corresponding to the original feature cluster is, and the smaller the possibility that the new standardized data belongs to this data type is. New standardized data and The larger the corrected membership between clusters, the greater the

[0084] In this way, the corrected membership between the centers of each feature clustering cluster of the new standardized data with respect to the multi-category data in the new multi-source heterogeneous data is obtained.

[0085] S600: Classify the new multi-source heterogeneous data into data types according to the modified membership degree.

[0086] After obtaining the revised membership between the centers of each feature cluster of the new standardized data with respect to the multi-category data in the newly acquired real-time multi-source heterogeneous data, the revised main feature cluster of each new standardized data can be confirmed again, and the new standardized data can be divided into the data type corresponding to the revised main feature cluster of the new standardized data. Therefore, according to the revised membership, the data type of the new multi-source heterogeneous data is divided, including: setting a membership threshold (which can be 90% according to experience), recording the original feature cluster corresponding to the revised membership of the new standardized data greater than the membership threshold as the revised feature cluster of the new standardized data; recording the original feature cluster corresponding to the maximum revised membership of the new standardized data as the revised main feature cluster of the new standardized data; and dividing the new standardized data into the data type to which the corresponding revised main feature cluster belongs.

[0087] In the historical multi-source heterogeneous data, the data types have been divided according to the main feature clustering clusters of each standardized data. The single-category data in the new multi-source heterogeneous data acquired in real time is directly divided into the data type corresponding to the original main feature clustering cluster; the multi-category data is divided into the data type divided in the historical multi-source heterogeneous data by the cluster corresponding to the largest modified membership (i.e., modified main feature clustering cluster), thus completing the division of the data types corresponding to each real-time multi-source heterogeneous data and the historical multi-source heterogeneous data.

[0088] Finally, the data is classified, stored and dynamically adjusted. According to the data type of each real-time multi-source heterogeneous data, the multi-source heterogeneous data is divided into corresponding storage nodes. Each storage node is organized according to different data types to ensure clear data classification and orderly storage; for single-category data, the corresponding storage node is relatively fixed because this type of data has a clear category affiliation; for multi-category data, after correction, it is stored in the storage node of the most suitable main feature clustering cluster.

[0089] As time goes by, the clustering results of the system's existing data are regularly checked to determine whether the cluster center needs to be updated. For example, when the amount of new data increases or the characteristics of the data change, the original cluster center may no longer be accurate. At this time, the cluster needs to be recalculated and the storage node needs to be updated.

[0090] Dynamically adjust based on the usage of storage nodes. For data types with higher access frequency, more storage resources are allocated first, while for historical multi-source heterogeneous data types with less access, they are migrated to low-frequency storage areas to save high-performance storage space.

[0091] Scan data storage nodes regularly to find duplicate or redundant data. Some of the historical multi-source heterogeneous data may be outdated or no longer active. Mark it and transfer it to the backup storage node to free up valuable storage space.

[0092] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0093] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A multi-source heterogeneous data integration storage method for traffic engineering construction, characterized in that: The method comprises: Collecting multi-source heterogeneous data during the construction of a traffic project, and performing standardization processing on the multi-source heterogeneous data to obtain standardized data; Performing feature extraction on the standardized data to obtain a feature vector for each piece of the standardized data; According to the feature vector, cluster analysis is performed on the standardized data, and based on the clustering result, membership analysis is performed on the standardized data to obtain a feature clustering cluster and a main feature clustering cluster of each piece of the standardized data; For historical multi-source heterogeneous data, data types are divided according to the main feature clustering clusters; For the new multi-source heterogeneous data acquired in real time, analyzing the distribution characteristics of the standardized data in the feature clustering cluster, and combining the membership degree to obtain the modified membership degree between the new multi-source heterogeneous data and each of the feature clustering cluster centers; When the feature cluster corresponding to a certain piece of new standardized data in the new multi-source heterogeneous data acquired in real time only contains the main feature cluster, it is determined to be single-category data; According to the modified membership, the new multi-source heterogeneous data except the single-category data is divided into data types to obtain modified main feature clusters; the new standardized data is divided into storage nodes corresponding to the corresponding main feature clusters for storage; For the new multi-source heterogeneous data acquired in real time, the distribution characteristics of the standardized data in the feature clustering cluster are analyzed, and the modified membership between the new multi-source heterogeneous data and each feature clustering cluster center is obtained by combining the membership, including: For the new multi-source heterogeneous data acquired in real time, standardization processing and feature extraction are first performed, and then membership analysis is performed to obtain the original membership and original feature clustering clusters corresponding to each new standardized data in the new multi-source heterogeneous data; In combination with the original membership, analyzing the proportional relationship between the amount of historical standardized data in the feature clustering cluster and the total amount of historical standardized data in all original feature clustering clusters corresponding to the new standardized data, to obtain the distribution characteristics of the standardized data in the feature clustering cluster; Using the distribution features as weights, weighted processing is performed on the original membership to obtain a modified membership between the new standardized data in the new multi-source heterogeneous data and each of the feature cluster centers; Combined with the original membership, the proportional relationship between the amount of historical standardized data in the feature clustering cluster and the total amount of historical standardized data in all original feature clustering clusters corresponding to the new standardized data is analyzed to obtain the distribution characteristics of the standardized data in the feature clustering cluster, including: Obtaining the number of historical standardized data in the feature clustering cluster; Get the total number of historical standardized data in all original feature clusters corresponding to the new standardized data; Using the original membership as a weight, processing the total quantity to obtain a weighted total quantity; Calculating the proportional relationship between the quantity and the weighted total quantity to obtain the distribution characteristics of the standardized data in the feature clustering cluster; The calculation formula for the modified membership between the new standardized data and each feature cluster center in constructing new multi-source heterogeneous data is: In the formula, Indicates New standardized data and The modified membership between clusters; To indicate the New standardized data and The original membership between clusters; Indicates The number of historical standardized data in the clusters; Indicates The number of clusters contained in the new standardized data, that is, The number of original feature clusters of the new standardized data; Indicates New standardized data and The original membership between the original feature clusters; For the New standardized data The number of historical standardized data in the original feature clusters.

2. The multi-source heterogeneous data integration and storage method for traffic engineering construction according to claim 1 is characterized in that: The multi-source heterogeneous data are subjected to standardization processing, and before that, the method further includes: performing data cleaning on the multi-source heterogeneous data.

3. The multi-source heterogeneous data integration and storage method for traffic engineering construction according to claim 2 is characterized in that: The multi-source heterogeneous data are subjected to standardization processing, including: converting all the multi-source heterogeneous data into decimal data to obtain standardized data.

4. The multi-source heterogeneous data integration and storage method for traffic engineering construction according to claim 1 is characterized in that: Performing feature extraction on the standardized data to obtain a feature vector of each piece of the standardized data includes: Extracting data features of each piece of the standardized data, wherein the data features include the number of peaks, the maximum peak, and the time difference between two adjacent peaks; The data features are integrated into a feature vector as a feature vector of each piece of the standardized data.

5. The multi-source heterogeneous data integration and storage method for traffic engineering construction according to claim 1 is characterized in that: Performing cluster analysis on the standardized data according to the feature vector, including: According to the feature vector, the elbow method is used to determine the K value for K-means clustering of each of the standardized data; According to the K value, the standardized data is clustered by a fuzzy C-means clustering algorithm to obtain K clusters for each piece of the standardized data.

6. The multi-source heterogeneous data integration and storage method for traffic engineering construction according to claim 5 is characterized in that: Based on the clustering results, the standardized data is subjected to membership analysis to obtain a feature clustering cluster and a main feature clustering cluster of each piece of the standardized data, including: Obtaining the Euclidean distance from the standardized data to the center of each cluster; Analyzing the distance ratio relationship between the Euclidean distance from the standardized data to the cluster center and the sum of the Euclidean distances to all the cluster centers to obtain the membership degree between the standardized data and the cluster center; Setting a membership threshold, recording the cluster whose membership corresponding to the standardized data is greater than the cluster corresponding to the membership threshold as a characteristic cluster of the standardized data; The cluster corresponding to the maximum membership degree corresponding to the standardized data is recorded as the main feature cluster of the standardized data.

7. The multi-source heterogeneous data integration and storage method for traffic engineering construction according to claim 1 is characterized in that: For historical multi-source heterogeneous data, data types are divided according to the main feature clusters, including: For historical multi-source heterogeneous data, data with the same main feature cluster corresponding to the standardized data are divided into one data type.

8. The multi-source heterogeneous data integration and storage method for traffic engineering construction according to claim 1 is characterized in that: According to the modified membership degree, the new multi-source heterogeneous data is divided into data types, including: Setting a membership threshold, recording the original feature cluster whose revised membership corresponding to the new standardized data is greater than the original feature cluster corresponding to the membership threshold as the revised feature cluster of the new standardized data; The original feature cluster corresponding to the maximum modified membership degree corresponding to the new standardized data is recorded as the modified main feature cluster of the new standardized data; The newly standardized data is divided into the data type to which the corresponding modified main feature cluster belongs.

Images