Multi-source heterogeneous test detection data analysis and treatment method

Abstract

The invention relates to the technical field of data processing, in particular to a multi-source heterogeneous test detection data analysis and treatment method. The method comprises the following steps: acquiring various detection data of the traffic infrastructure; obtaining the uniqueness of each type of detection data according to the change difference and the correlation condition of each type of detection data and other types of detection data of the same type and the distribution condition of each type of detection data; according to the similar condition of the attribute information of each type of detection data and the same type of unstructured data, obtaining the attribute density of each type of detection data; according to the uniqueness and the attribute density, obtaining the importance degree of each type of detection data; and optimizing the PCA dimension reduction algorithm according to the importance degree, and obtaining the detection data after optimization and dimension reduction. According to the method, the PCA dimension reduction algorithm is optimized by obtaining the importance degree of each detection data, the loss rate of important data in the dimension reduction process is reduced, and the analysis precision of the data after dimension reduction is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for analyzing and managing multi-source heterogeneous test detection data. Background Art

[0002] As digital transformation deepens, enterprises and society generate a large amount of data from diverse sources and structures, forming the so-called "multi-source heterogeneous data". Multi-source refers to data sources with significant differences, such as data from sensors, social media, databases, and documents; heterogeneous refers to different data structures, such as different probability distributions, densities, and attributes of different types of data. Multi-source heterogeneous data sources generally refer to data from different sources, in different forms, structures, or using different standards.

[0003] In the process of traffic infrastructure quality inspection, the integration and analysis of multi-source heterogeneous test data is the key to ensure the comprehensiveness and accuracy of infrastructure performance evaluation. Multi-source heterogeneous test data comes from multiple data acquisition channels such as different monitoring equipment and management systems. Multi-source heterogeneous test data is usually very large. If the multi-source heterogeneous test data is directly integrated and analyzed, it will cause a high computational cost. At the same time, it is difficult to accurately analyze the multi-source heterogeneous test data, and thus the quality of traffic infrastructure cannot be accurately detected. In the existing method, the multi-source heterogeneous test data is reduced in dimension by the PCA (principal component analysis) dimensionality reduction algorithm, and the quality of traffic infrastructure is accurately and efficiently monitored by the multi-source heterogeneous test data after dimensionality reduction. However, in actual situations, the PCA (principal component analysis) dimensionality reduction algorithm only performs dimensionality reduction analysis based on the distribution characteristics of each multi-source heterogeneous test data. Some important information may be lost in the dimensionality reduction process, resulting in the multi-source heterogeneous test data after dimensionality reduction being unable to more accurately analyze the quality of traffic infrastructure. Summary of the invention

[0004] In order to solve the technical problem that the PCA dimensionality reduction algorithm only performs dimensionality reduction analysis through the distribution characteristics of each multi-source heterogeneous test detection data, some important information may be lost in the dimensionality reduction process, resulting in the multi-source heterogeneous test detection data after dimensionality reduction cannot more accurately analyze the quality of transportation infrastructure, the purpose of the present invention is to provide a multi-source heterogeneous test detection data analysis and management method, and the technical scheme adopted is as follows:

[0005] The embodiment of the present invention provides a multi-source heterogeneous test detection data analysis and management method, the method comprising the following steps:

[0006] Obtain various detection data of transportation infrastructure within a preset time period;

[0007] Classify all test data into types, and obtain the uniqueness of each test data according to the change difference and correlation between each test data and other test data of the same type, as well as the distribution of each test data;

[0008] According to the similarity between the attribute information of each test data and the unstructured data of the same type, the attribute density of each test data is obtained;

[0009] Obtain the importance of each test data according to its uniqueness and attribute density;

[0010] The PCA dimensionality reduction algorithm is optimized according to the importance level to obtain the detection data after optimized dimensionality reduction.

[0011] Furthermore, the method for obtaining the uniqueness is:

[0012] According to the difference between each test data and other test data of the same type, the sensitivity of each test data is obtained;

[0013] Obtaining the correlation degree of each test data according to the Pearson correlation coefficient between each test data and each other test data of the same type within a preset time period;

[0014] According to the distribution of each test data within a preset time period, obtain the degree of change of each test data;

[0015] The uniqueness of each detection data is obtained according to its sensitivity, correlation and variation; wherein the sensitivity and variation are positively correlated with the uniqueness, and the correlation is negatively correlated with the uniqueness.

[0016] Furthermore, the sensitivity is obtained by:

[0017] For any type of detection data, the detection data within a preset time period is plotted as a broken line according to the time sequence, and an upper envelope and a lower envelope of the broken line are obtained;

[0018] Obtaining the absolute values ​​of the tangent slopes of the upper envelope and the lower envelope at each point of the fold line in time sequence, and sequentially constructing an upper envelope slope sequence and a lower envelope slope sequence of the detection data;

[0019] The average of the absolute values ​​of the tangent slopes in the same order in all upper envelope slope sequences of the type to which the detection data belongs is used as the reference value in the corresponding order of the upper envelope slope sequence of the type to which the detection data belongs;

[0020] The average of the absolute values ​​of the tangent slopes in the same order in all the lower envelope slope sequences of the type to which the detection data belongs is used as the reference value in the corresponding order of the lower envelope slope sequence of the type to which the detection data belongs;

[0021] For any order, the difference between the absolute value of the tangent slope of the upper envelope slope sequence of the detection data in the order and the reference value of the upper envelope slope sequence of the type to which the detection data belongs in the order is taken as the discrete degree of the absolute value of the tangent slope of the upper envelope slope sequence of the detection data in the order;

[0022] The difference between the absolute value of the tangent slope of the lower envelope slope sequence of the detection data in the order and the reference value of the lower envelope slope sequence of the type to which the detection data belongs in the order is used as the discrete degree of the absolute value of the tangent slope of the lower envelope slope sequence of the detection data in the order;

[0023] The sum of the discrete degrees of the absolute values ​​of the tangent slopes of the upper envelope slope sequence and the lower envelope slope sequence of the detection data in the order is taken as the overall discrete degree of the detection data in the order;

[0024] The sensitivity of the detection data is obtained according to the overall discreteness of the detection data in each order; wherein the overall discreteness is positively correlated with the sensitivity.

[0025] Furthermore, the method for obtaining the degree of change is:

[0026] For any type of detection data, the reference values ​​of each order of the upper envelope slope sequence in the type to which the detection data belongs are respectively negatively correlated and normalized, and the results are all used as the first correction weight;

[0027] Correcting the variance and mean of the absolute value of the tangent slope in the upper envelope slope sequence of the detection data by using the first correction weight, respectively, to obtain the corrected variance and corrected mean of the upper envelope slope sequence of the detection data;

[0028] The reference values ​​of each order of the lower envelope slope sequence of the type to which the detection data belongs are respectively negatively correlated and normalized, and the results are all used as the second correction weight;

[0029] Correcting the variance and mean of the absolute value of the tangent slope in the lower envelope slope sequence of the detection data by using the second correction weight, respectively, to obtain the corrected variance and corrected mean of the lower envelope slope sequence of the detection data;

[0030] The mean of the corrected variances of the upper envelope slope sequence and the lower envelope slope sequence of the detection data is taken as the overall fluctuation value;

[0031] The average of the corrected mean values ​​of the upper envelope slope sequence and the lower envelope slope sequence of the detection data is taken as the overall change rate value;

[0032] The degree of change of the detection data is obtained according to the overall fluctuation value and the overall change rate value; wherein the overall fluctuation value and the overall change rate value are both positively correlated with the degree of change.

[0033] Furthermore, the calculation formulas for the corrected variance and corrected mean of the upper envelope slope sequence are:

[0034] The calculation formula of the corrected variance is: The calculation formula of the modified mean is: In the formula, is the corrected variance of the upper envelope slope sequence A of the k-th detection data; I is the total number of elements in the upper envelope slope sequence A of the k-th detection data; is the reference value of the upper envelope slope sequence A of the i-th order in the type to which the k-th detection data belongs; norm is the normalization function; is the first modified weight; a k,i is the absolute value of the i-th tangent slope in the upper envelope slope sequence A of the k-th detection data; is the mean of the absolute values ​​of all tangent slopes in the upper envelope slope sequence A of the k-th detection data; is the corrected mean of the upper envelope slope sequence A of the k-th detection data.

[0035] Furthermore, the method for obtaining the attribute density is:

[0036] For any type of detection data, obtain the attribute information of the detection data and the word vector of the unstructured data of the same type as the detection data;

[0037] For any word vector corresponding to the attribute information of the detection data, obtain the mean of the modified cosine similarity between the word vector and each word vector of the unstructured data of the same type as the detection data as the first eigenvalue of the word vector;

[0038] The average of the first eigenvalues ​​of all word vectors corresponding to the attribute information of the detection data is used as the attribute density value of the detection data.

[0039] Furthermore, the method for obtaining the importance is:

[0040] The result of normalizing the product of the uniqueness value and the attribute density value of each detection data is taken as the importance value of each detection data.

[0041] Furthermore, the method of optimizing the PCA dimensionality reduction algorithm according to the importance and obtaining the detection data after optimization and dimensionality reduction is:

[0042] For any type, construct the detection data of the type into an initial matrix, wherein one row of the initial matrix corresponds to one type of detection data;

[0043] Each row of data in the initial matrix is ​​processed by zero mean to obtain a first process matrix;

[0044] Multiply the importance value of each detection data of the type by each element in the corresponding row of the first process matrix to obtain a second process matrix;

[0045] Obtaining eigenvalues ​​and eigenvectors of the covariance matrix of the second process matrix, and arranging the eigenvectors from top to bottom in rows according to the order of corresponding eigenvalues ​​from large to small to form a third process matrix;

[0046] The matrix consisting of the first preset number of rows in the third process matrix is ​​used as a dimension reduction matrix;

[0047] The initial matrix is ​​multiplied by the dimension reduction matrix to obtain an optimized dimension reduction matrix; the data in the optimized dimension reduction matrix is ​​the detection data after optimized dimension reduction.

[0048] Furthermore, the method for obtaining the preset number is:

[0049] Obtain the cumulative contribution rate of each eigenvalue, and add up the cumulative contribution rates from large to small. When the cumulative result is greater than the preset cumulative contribution rate threshold, stop adding up.

[0050] The number of eigenvalues ​​corresponding to the accumulation process is used as the preset number.

[0051] Furthermore, the method for classifying all detection data into types is:

[0052] All detection data are divided into types according to different collection methods.

[0053] The present invention has the following beneficial effects:

[0054] All detection data are divided into types to improve the accuracy of analyzing the relationship between data, which is conducive to the subsequent accurate acquisition of the importance of each detection data; in order to analyze the importance of each detection data, the uniqueness of each detection data is obtained according to the change differences and related situations between each detection data and other detection data of the same type, as well as the distribution of each detection data, to accurately reflect the degree to which each detection data can be replaced, and preliminarily determine the importance of each detection data; further, according to the similarity between the attribute information of each detection data and the unstructured data of the same type, the attribute density of each detection data is obtained, accurately reflecting the importance of the attributes of each detection data, and further determining the importance of each detection data; and then according to the uniqueness and attribute density of each detection data, the importance of each detection data is accurately obtained to avoid the loss rate of important data in the process of data dimensionality reduction, and then the PCA dimensionality reduction algorithm is optimized according to the importance, the rationality of the PCA dimensionality reduction algorithm is improved, and the detection data after optimization and dimensionality reduction is accurately obtained, and then the quality of transportation infrastructure is accurately and efficiently analyzed based on the detection data after optimization and dimensionality reduction, which is conducive to timely maintenance of transportation infrastructure. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic flow chart of a multi-source heterogeneous test detection data analysis and management method provided by an embodiment of the present invention;

[0057] Figure 2 A flow chart of a method for obtaining a unique degree provided by one embodiment of the present invention;

[0058] Figure 3 A structural diagram of a multi-source heterogeneous test detection data analysis and management system provided by an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, characteristics and effects of a multi-source heterogeneous test detection data analysis and management method proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0061] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0062] The following is a detailed description of a multi-source heterogeneous test data analysis and management method provided by the present invention in conjunction with the accompanying drawings.

[0063] Embodiment 1:

[0064] The specific scenario of this embodiment is: in the process of traffic infrastructure quality inspection, a large number of multi-source heterogeneous test data with rich types will be generated, mainly including structured and unstructured data such as sensor monitoring data, geographic information data, physical test data, environmental monitoring data and image video data. In order to evaluate the health status and safety level of traffic infrastructure and accurately predict the maintenance needs of traffic infrastructure, it is necessary to fuse the multi-source heterogeneous test data generated in the process of traffic infrastructure quality inspection. In the data fusion process, due to the large amount of multi-source heterogeneous test data, it is particularly easy to cause inaccurate data fusion and consume a lot of cost. In order to make the data easier to process and highlight more important feature information, the PCA dimensionality reduction algorithm is used to reduce the dimensionality of the multi-source heterogeneous test data. However, in actual situations, the PCA dimensionality reduction algorithm only processes the distribution characteristics of the multi-source heterogeneous test data, and fails to consider the importance of each multi-source heterogeneous test data to the quality inspection of traffic infrastructure, which may lead to the loss of some important information. Therefore, this embodiment first divides all the detection data into types, and then obtains the uniqueness of each detection data based on the change differences, related situations, and distribution of each detection data of the same type and different types, and preliminarily analyzes the degree of retention of each detection data in the dimensionality reduction process, and further obtains the attribute density of each detection data based on the similarity between the attribute information of each detection data and the unstructured data of the same type, analyzes the criticality of each detection data in characterizing the quality of transportation infrastructure, and then obtains the importance of each detection data based on the uniqueness and attribute density of each detection data, determines the retention degree of each detection data in the PCA dimensionality reduction algorithm, reduces the loss rate of important detection data in the dimensionality reduction process, improves the analysis accuracy of the detection data after dimensionality reduction, and then accurately detects the quality of transportation infrastructure. Among them, the PCA dimensionality reduction algorithm is a well-known technology and will not be described in detail.

[0065] This invention proposes a multi-source heterogeneous test data analysis and management method, please refer to Figure 1 , which shows a schematic flow chart of a multi-source heterogeneous test detection data analysis and management method provided by an embodiment of the present invention, the method comprising the following steps:

[0066] Step S1: Acquire various detection data of traffic infrastructure within a preset time period.

[0067] Specifically, in order to detect the quality of transportation infrastructure and ensure the safety of transportation infrastructure, this embodiment deploys sensors and other detection equipment to obtain various detection data within a preset time period of transportation infrastructure, wherein the detection data includes structured and unstructured data such as various sensor monitoring data, geographic information data, physical test data, environmental monitoring data, and image and video data. The detection data is multi-source heterogeneous test detection data. Processing the detection data is a systematic process, which aims to unify the detection data of different sources, structures and formats to support efficient analysis and decision-making. In order to conduct real-time analysis of the quality of transportation infrastructure, this embodiment sets the preset time period to half a year and the end time of the preset time period to the current time. At the same time, the time interval between adjacent times of data collection is set to 24 hours. The implementer can set the size of the time interval between the preset time period and the adjacent time according to the actual situation, which is not limited here.

[0068] In order to improve the accuracy of the subsequent analysis process, this embodiment first cleans each type of test data obtained, wherein the cleaning includes operations such as missing value processing, deduplication, sorting, and outlier processing. Then, each type of cleaned test data is standardized to eliminate the impact of different test data dimensions on the analysis. Among them, cleaning and standardization are both well-known technologies and will not be described in detail. It should be noted that the test data that appear subsequently are all cleaned and standardized test data.

[0069] Step S2: Classify all the detection data into types, and obtain the uniqueness of each detection data according to the change difference and correlation between each detection data and other detection data of the same type, as well as the distribution of each detection data.

[0070] Specifically, the inspection conditions of different transportation infrastructure construction projects are different, and the collected inspection data are also different, that is, there are differences in the attributes, structures, forms and other characteristics of various inspection data. In order to increase the rationality of subsequent inspection data feature analysis, all inspection data are classified according to different collection methods. In this embodiment, the inspection data is divided into time type data and space type data, among which, time type data refers to data distinguished by time parameters, such as moment, time period, etc.; space type data refers to data distinguished by space parameters, such as position, coordinates, etc.

[0071] In order to accurately and efficiently integrate the detection data and then accurately analyze the quality of the transportation infrastructure, it is necessary to perform dimensionality reduction processing on the detection data. In order to improve the rationality of the dimensionality reduction of the detection data, this embodiment performs dimensionality reduction processing on the time type data and the space type data separately. Among them, the process of dimensionality reduction processing for the time type data and the space type data is consistent. The detection data corresponding to different construction projects and detection processes have different expressiveness on the quality information representing the transportation infrastructure. In the dimensionality reduction process, it is necessary to analyze the expressiveness of each detection data, optimize the dimensionality reduction steps, and improve the rationality and accuracy of the dimensionality reduction process. In actual situations, when the changes in a certain detection data are more unique and less affected by other detection data of the same type, it means that the detection data is more unique, the transportation infrastructure information represented by the detection data is more difficult to be replaced by other types of detection data, the detection data is more important, and the information retained in the dimensionality reduction process is more. It is known that when the difference between a certain detection data and other detection data of the same type is greater, it means that the detection data is more unique; when the detection data is less correlated with other detection data of the same type, it also means that the detection data is more unique; when the change of the detection data is more unstable, it means that the change of the detection data is more special, which can also reflect that the detection data is more unique. Therefore, this embodiment obtains the uniqueness of each detection data according to the difference and correlation between each detection data and other detection data of the same type, as well as the distribution of each detection data. The greater the uniqueness, the more irreplaceable the traffic infrastructure information represented by the corresponding type of detection data, and the more important the corresponding type of detection data.

[0072] Preferably, in one possible implementation of this embodiment, the method for obtaining the uniqueness can refer to Figure 2 , which shows a flow chart of a method for obtaining a unique degree provided by this embodiment, the method comprising the following steps:

[0073] Step S201: Obtain the sensitivity of each detection data according to the difference between each detection data and other detection data of the same type.

[0074] When the difference between a certain test data and other test data of the same type is greater, it means that the change perception ability of this test data is more sensitive and the test data is more special. Therefore, according to the difference between each test data and other test data of the same type, the sensitivity of each test data is obtained. The greater the sensitivity, the more special the corresponding type of test data.

[0075] In one possible implementation of the present embodiment, the method for obtaining the sensitivity is as follows: for any type of detection data, the detection data within a preset time period is plotted as a broken line according to the time sequence. In order to accurately analyze the change of the detection data, the present embodiment obtains the upper envelope and the lower envelope of the broken line, and reflects the change of the detection data by analyzing the change of the upper envelope and the lower envelope. Among them, the method for obtaining the envelope is a well-known technology and will not be described in detail. According to the time sequence, the absolute values ​​of the tangent slopes of the upper envelope and the lower envelope at each point of the broken line are respectively obtained, and the upper envelope slope sequence and the lower envelope slope sequence of the detection data are sequentially constructed; for example, taking the upper envelope of the broken line corresponding to the detection data as an example, the absolute values ​​of the tangent slopes of the points on the broken line through which the upper envelope passes are arranged according to the time sequence of the corresponding points from front to back, which is the upper envelope slope sequence of the detection data. Similarly, the lower envelope slope sequence of the detection data is obtained. The method for obtaining the tangent slope is a well-known technique and will not be described in detail.

[0076] For the same type of data, the elements in the same order in the upper envelope slope sequence of various tests, i.e., the absolute values ​​of the tangent slopes, have a certain connection. For time type data, the elements in the same order in the upper envelope slope sequence of various tests represent the changes in the test data at the same or close time nodes; for space type data, the elements in the same order in the upper envelope slope sequence of various tests represent the changes in the test data at the same or close spatial positions. It should be noted that the lengths of the upper envelope slope sequences of various tests of the same type of data are the same. The above analysis is also applicable to the lower envelope slope sequence;

[0077] For obtaining the lower envelope slope sequence from the upper envelope slope sequence, the elements in a certain order correspond to the numerical fluctuation rate of the actual detection data. The larger the element in the order, the higher the probability of an "event" occurring at the current time point. For example, when a traffic lane line cracks or fades at a certain time point, multiple sensors (such as flatness detection and lane line quality detection, etc.) will change, that is, for time type data, an "event" occurs in the corresponding order at that time point;

[0078] Among them, different types of detection data have different response sensitivities to "events". In order to obtain the response sensitivity of this type of detection data to "events", the average of the absolute values ​​of the tangent slopes in the same order of all upper envelope slope sequences in the type to which this type of detection data belongs is used as the reference value in the corresponding order of the upper envelope slope sequence of the type to which this type of detection data belongs; the average of the absolute values ​​of the tangent slopes in the same order of all lower envelope slope sequences in the type to which this type of detection data belongs is used as the reference value in the corresponding order of the lower envelope slope sequence of the type to which this type of detection data belongs; it should be noted that the order of this embodiment refers to the order of the positions of the elements in the envelope slope sequence in the envelope slope sequence; and for any order, the absolute value of the tangent slope of the upper envelope slope sequence of this type of detection data in that order is compared with the absolute value of the tangent slope of the upper envelope slope sequence of this type of detection data. The absolute value of the difference between the reference values ​​of the upper envelope slope sequence of the type in this order is used as the discrete degree value of the absolute value of the tangent slope of the upper envelope slope sequence of the detection data in this order; the absolute value of the difference between the absolute value of the tangent slope of the lower envelope slope sequence of the detection data in this order and the reference value of the lower envelope slope sequence of the type to which the detection data belongs in this order is used as the discrete degree value of the tangent slope of the lower envelope slope sequence of the detection data in this order; wherein, the larger the discrete degree value, the more obvious the difference between the change of the detection data in this order at the time point and the change of other detection data of the same type, and then the sum of the discrete degree values ​​of the absolute values ​​of the tangent slope of the upper envelope slope sequence and the lower envelope slope sequence of the detection data in this order is used as the overall discrete degree of the detection data in this order;

[0079] In order to more accurately represent the difference between the detection data and other detection data, that is, to reflect the sensitivity of the detection data, the sensitivity of the detection data is obtained according to the overall discreteness of the detection data in each order; wherein the overall discreteness is positively correlated with the sensitivity. In this embodiment, the sum of the overall discreteness of the detection data in all orders is used as the sensitivity value of the detection data.

[0080] At this point, the sensitivity value of each detection data is obtained.

[0081] Step S202: Obtain the correlation degree of each test data according to the Pearson correlation coefficient between each test data and each other test data of the same type within a preset time period.

[0082] When the absolute value of the Pearson correlation coefficient between a certain detection data and each other detection data of the same type is larger, it means that the correlation degree of this detection data with other detection data is greater, the traffic infrastructure information represented by this detection data is more easily replaced by other detection data, and this detection data is less unique. Therefore, according to the Pearson correlation coefficient of each detection data with each other detection data of the same type within a preset time period, the correlation degree of each detection data is obtained. The greater the correlation degree, the less unique the corresponding type of detection data is. Among them, the method for obtaining the Pearson correlation coefficient is a well-known technology and will not be repeated here.

[0083] In one possible implementation of this embodiment, the method for obtaining the degree of correlation is: for any type of detection data, obtain the Pearson correlation coefficient of the detection data and each other detection data of the same type within a preset time period, all of which are used as reference correlation values; and use the absolute value of the mean of the reference correlation values ​​as the degree of correlation value of the detection data. The smaller the degree of correlation value, the more unique the detection data.

[0084] At this point, the correlation value of each detection data is obtained.

[0085] Step S203: Obtain the variation degree of each detection data according to the distribution of each detection data in a preset time period.

[0086] When the absolute values ​​of the tangent slopes in the upper envelope slope sequence and the lower envelope slope sequence of a certain detection data are larger and the fluctuation of the absolute values ​​of the tangent slopes is also larger, it means that the degree of change of this detection data is greater, and the detection data is more likely to be special. It is known that when an "event" occurs in the detection process, the probability of changes in various detection data is higher. At this time, the change of the detection data itself analyzed by the envelope slope sequence is inaccurate. Therefore, it is necessary to correct the variance and mean of the absolute values ​​of the tangent slopes in the envelope slope sequence of each detection data in combination with the probability of event occurrence, obtain the corrected variance and corrected mean of the envelope slope sequence, and then accurately obtain the degree of change of each detection data. The greater the degree of change, the more special the corresponding type of detection data.

[0087] In one possible implementation of this embodiment, the method for obtaining the degree of change is as follows: for any type of detection data, the reference values ​​in each order of the upper envelope slope sequence in the type to which the detection data belongs are respectively negatively correlated and normalized, and all are used as the first correction weight; in actual situations, the larger the reference value, the greater the probability of the existence of an "event", therefore, the larger the reference value, the smaller the first correction weight, reducing the degree of influence of the "event" on the change of the detection data itself. Then, the variance and mean of the absolute value of the tangent slope in the upper envelope slope sequence of the detection data are respectively corrected by the first correction weight to obtain the corrected variance and corrected mean of the upper envelope slope sequence of the detection data;

[0088] The calculation formula of the corrected variance is: The calculation formula for the modified mean is: In the formula, is the corrected variance of the upper envelope slope sequence A of the k-th detection data; I is the total number of elements in the upper envelope slope sequence A of the k-th detection data; is the reference value of the upper envelope slope sequence A of the i-th order in the type to which the k-th detection data belongs; norm is the normalization function; is the first modified weight; a k,i is the absolute value of the i-th tangent slope in the upper envelope slope sequence A of the k-th detection data; is the mean of the absolute values ​​of all tangent slopes in the upper envelope slope sequence A of the k-th detection data; is the corrected mean of the upper envelope slope sequence A of the k-th detection data;

[0089] Similarly, the same analysis is performed on the lower envelope slope sequence of the type to which the detection data belongs: the reference values ​​of each order of the lower envelope slope sequence of the type to which the detection data belongs are negatively correlated and normalized, and the results are all used as the second correction weights; the variance and mean of the absolute value of the tangent slope in the lower envelope slope sequence of the detection data are corrected by the second correction weights, respectively, to obtain the corrected variance and corrected mean of the lower envelope slope sequence of the detection data;

[0090] The calculation formula of the corrected variance is: The calculation formula for the modified mean is: In the formula, is the corrected variance of the lower envelope slope sequence B of the k-th detection data; N is the total number of elements in the lower envelope slope sequence B of the k-th detection data; is the reference value of the nth order of the lower envelope slope sequence B in the type to which the kth detection data belongs; norm is the normalization function; is the second modified weight; b k,nis the absolute value of the nth tangent slope in the lower envelope slope sequence B of the kth detection data; is the mean of the absolute values ​​of all tangent slopes in the lower envelope slope sequence B of the k-th detection data; is the corrected mean of the lower envelope slope sequence B of the k-th detection data;

[0091] In order to accurately express the change of this kind of detection data, the mean of the corrected variance of the upper envelope slope sequence and the lower envelope slope sequence of this kind of detection data is further used as the overall fluctuation value; the larger the overall fluctuation value, the greater the difference in the local change rate of this kind of detection data after excluding the influence of the "event", and the greater the change of this kind of detection data itself; the mean of the corrected mean of the upper envelope slope sequence and the lower envelope slope sequence of this kind of detection data is used as the overall change rate value; the larger the overall change rate value, the greater the local change rate of this kind of detection data after excluding the influence of the "event", which also indicates that the change of this kind of detection data itself is greater. Then, according to the overall fluctuation value and the overall change rate value, the change degree of this kind of detection data is obtained; wherein, the overall fluctuation value and the overall change rate value are both positively correlated with the change degree. In this embodiment, the product of the overall fluctuation value and the overall change rate value is used as the change degree value of this kind of detection data. In other embodiments, the addition result of the overall fluctuation value and the overall change rate value can be used as the change degree value of this kind of detection data, which is not limited here.

[0092] At this point, the change degree value of each detection data is obtained.

[0093] Step S204: Obtain the uniqueness of each detection data according to the sensitivity, correlation and variation of each detection data; wherein the sensitivity and variation are both positively correlated with the uniqueness, and the correlation is negatively correlated with the uniqueness.

[0094] It is known that the greater the sensitivity, the more special the corresponding type of detection data; the greater the correlation, the easier it is for the traffic infrastructure information represented by the corresponding type of detection data to be replaced by other detection data, and the less unique the detection data is; the greater the degree of variation, the more special the characteristics of the corresponding type of detection data itself, and the more unique the corresponding type of detection data is. Therefore, this embodiment obtains the uniqueness of each type of detection data based on the sensitivity, correlation and variation of each type of detection data; wherein the sensitivity and variation are both positively correlated with the uniqueness, and the correlation is negatively correlated with the uniqueness.

[0095] In order to accurately represent the uniqueness of each type of detection data, this embodiment specifically quantifies the uniqueness as a uniqueness value. The larger the uniqueness value, the less likely the traffic infrastructure information represented by the corresponding type of data is to be replaced. The more important the corresponding type of data is, the more information is retained during the dimensionality reduction process. The calculation formula for the uniqueness value is: Where U k is the uniqueness value of the k-th detection data; S k is the sensitivity value of the k-th detection data; X k is the correlation value of the kth detection data, γ is the first preset constant, which is greater than 0; J k is the change degree value of the k-th detection data.

[0096] In this embodiment, γ is set to 0.1 to avoid the denominator being 0. The implementer can set the value of γ according to the actual situation, which is not limited here. Get U k , which is not limited here.

[0097] At this point, the uniqueness value of each detection data is obtained.

[0098] Step S3: According to the similarity between the attribute information of each detection data and the unstructured data of the same type, the attribute density of each detection data is obtained.

[0099] Specifically, when the frequency of occurrence of the attribute information of a certain detection data in the same type of unstructured data is higher, it means that the detection data is more important, the detection data represents more information about the traffic infrastructure, and the detection data should retain more components in the process of dimensionality reduction. Therefore, the more similar the attribute information of a certain detection data is to the unstructured data of the same type, the more important the detection data is. Furthermore, this embodiment obtains the attribute density of each detection data based on the similarity between the attribute information of each detection data and the unstructured data of the same type. The greater the attribute density, the more times the attribute of the corresponding detection data appears, and the more important the corresponding detection data is.

[0100] It is known that the attribute information of the detection data and the unstructured data are mainly text data such as reports. In order to analyze the similarity between the attribute information of each detection data and the unstructured data of the same type, therefore, this embodiment first uses a word segmentation algorithm, such as the jieba algorithm, to perform word segmentation on the attribute information of each detection data and the unstructured data of the same type, and then uses a word vector conversion algorithm, such as the word2vec algorithm, to convert the word segmentation into a word vector. Among them, the jieba algorithm and the word2vec algorithm are both well-known technologies and will not be repeated here.

[0101] Preferably, in a method that can be implemented in this embodiment, the method for obtaining the attribute density is: for any kind of detection data, the attribute information of the detection data and the word vector of the unstructured data of the same type to which the detection data belongs are obtained; for any word vector corresponding to the attribute information of the detection data, the mean of the modified cosine similarity between the word vector and each word vector of the unstructured data of the same type to which the detection data belongs is obtained, as the first eigenvalue of the word vector; the larger the first eigenvalue, the more times the word vector corresponding to the attribute information of the detection data appears, which indirectly indicates that the attribute information of the detection data is more widely distributed, and the information represented by the detection data is more important. Among them, the method for obtaining the modified cosine similarity is a well-known technology and will not be repeated. In order to more accurately obtain the distribution of the attribute information of the detection data, the mean of the first eigenvalues ​​of all word vectors corresponding to the attribute information of the detection data is used as the attribute density value of the detection data. The larger the attribute density value, the more relevant information about the traffic infrastructure represented by the detection data, and the more important the detection data.

[0102] At this point, the attribute density value of each detection data is obtained.

[0103] Step S4: Obtain the importance of each detection data according to the uniqueness and attribute density of each detection data.

[0104] Specifically, it is known that the greater the uniqueness, the more difficult it is to replace the traffic infrastructure information reflected by the corresponding type of detection data, and the more important the corresponding type of detection data; the greater the attribute density, the more important the attribute information of the corresponding type of detection data. Therefore, this embodiment obtains the importance of each type of detection data according to the uniqueness and attribute density of each type of detection data. Among them, the uniqueness and attribute density are both positively correlated with the importance. The greater the importance, the more corresponding detection data is retained in the dimensionality reduction process.

[0105] In one possible implementation of this embodiment, the importance is obtained by normalizing the product of the uniqueness value and the attribute density value of each detection data to obtain the importance value of each detection data. The calculation formula of the importance value is: k =norm(U k ×ρ k );where Z k is the importance value of the k-th detection data; U k is the unique degree value of the k-th detection data; ρ k is the attribute density value of the k-th detection data; norm is the normalization function. In other embodiments, norm(U k +ρk )Get Z k , which is not limited here.

[0106] At this point, the importance value of each detection data is obtained.

[0107] Step S5: Optimize the PCA dimensionality reduction algorithm according to the importance level to obtain the detection data after optimization and dimensionality reduction.

[0108] Specifically, it is known that the greater the importance, the more important the transportation infrastructure information represented by the corresponding type of detection data, and the more information the corresponding type of detection data should retain during the dimensionality reduction process. Therefore, this embodiment optimizes the PCA dimensionality reduction algorithm according to the importance, obtains the optimized and reduced detection data, and performs accurate and efficient analysis of the transportation infrastructure, thereby accurately analyzing the quality of the transportation infrastructure.

[0109] The PCA dimensionality reduction algorithm is optimized according to the importance, and the method for obtaining the optimized and reduced detection data is as follows: for any type, the detection data of this type is constructed into an initial matrix, wherein one row of the initial matrix corresponds to one detection data, that is, the number of rows of the initial matrix is ​​the number of types of detection data in this type, and the number of columns of the initial matrix is ​​the number of times data is collected within a preset time period. Each row of data in the initial matrix is ​​subjected to zero mean processing to obtain a first process matrix; wherein, the zero mean processing in the PCA dimensionality reduction algorithm is a well-known technology and will not be described in detail. In order to retain the detection data that is important for characterizing the information of the traffic infrastructure as much as possible during the dimensionality reduction process, this embodiment multiplies the importance value of each detection data in this type by each element in the corresponding row of the first process matrix to obtain a second process matrix; for example, taking the kth detection data as an example, if the kth detection data corresponds to the third row in the first process matrix, the importance value of the kth detection data is multiplied by each element in the third row in the first process matrix. In order to obtain the detection data after optimized dimensionality reduction, the eigenvalues ​​and eigenvectors of the covariance matrix of the second process matrix are further obtained, and the eigenvectors are arranged from top to bottom in rows according to the corresponding eigenvalues ​​from large to small to form a third process matrix. The cumulative contribution rate of each eigenvalue is obtained, and the cumulative contribution rates are accumulated from large to small. In this embodiment, the preset cumulative contribution rate threshold is set to 85%. The implementer can set the preset cumulative contribution rate threshold according to the actual situation, which is not limited here. When the accumulated result is greater than the preset cumulative contribution rate threshold, the accumulation is stopped; the number of eigenvalues ​​corresponding to the accumulation process is used as the preset number. Then, the matrix composed of the first preset number of rows in the third process matrix is ​​used as a dimensionality reduction matrix; the initial matrix is ​​multiplied by the dimensionality reduction matrix to obtain an optimized dimensionality reduction matrix; the data in the optimized dimensionality reduction matrix is ​​the detection data after optimized dimensionality reduction. Among them, the method for obtaining the cumulative contribution rate in the PCA dimensionality reduction algorithm is a well-known technology and will not be repeated.

[0110] For the detection data after optimization and dimensionality reduction, structured data can be stored and managed using relational databases, such as MySQL database and PostgreSQL database, etc., which are not limited here, to ensure clear associations between data tables; unstructured data can be stored in distributed file systems (such as Hadoop Distributed FileSystem) or object storage services, which are not limited here. In addition, a data synchronization strategy should be set up, considering the synchronization frequency and method between different data sources to ensure data consistency and accuracy; at the same time, an access control mechanism should be set up to ensure that only authorized users can access relevant data.

[0111] Data fusion technology is a technical means to integrate multi-source heterogeneous data to provide more comprehensive, accurate and reliable information. Therefore, this embodiment will optimize the detection data after dimensionality reduction for fusion processing, which can reduce the analysis and fusion costs, while accurately obtaining information on transportation infrastructure and accurately detecting the quality of transportation infrastructure.

[0112] In summary, this embodiment obtains various detection data of traffic infrastructure; obtains the uniqueness of each detection data according to the change differences and correlation between each detection data and other detection data of the same type, as well as the distribution of each detection data; obtains the attribute density of each detection data according to the similarity between the attribute information of each detection data and the unstructured data of the same type; obtains the importance of each detection data according to the uniqueness and attribute density; optimizes the PCA dimensionality reduction algorithm according to the importance, and obtains the detection data after optimized dimensionality reduction. The present invention optimizes the PCA dimensionality reduction algorithm by obtaining the importance of each detection data, thereby reducing the loss rate of important data in the dimensionality reduction process and improving the analysis accuracy of the data after dimensionality reduction.

[0113] Embodiment 2:

[0114] The present invention also proposes a multi-source heterogeneous test data analysis and management system, please refer to Figure 3 , which shows a structure diagram of a multi-source heterogeneous test detection data analysis and management system provided by an embodiment of the present invention, the system includes: a data acquisition module 10, a uniqueness acquisition module 20, an attribute density acquisition module 30, an importance acquisition module 40 and a data processing module 50.

[0115] The data acquisition module 10 is used to acquire various detection data of the traffic infrastructure within a preset time period.

[0116] The uniqueness acquisition module 20 is used to classify all the detection data into types, and acquire the uniqueness of each detection data according to the change difference and correlation between each detection data and other detection data of the same type, as well as the distribution of each detection data.

[0117] The attribute density acquisition module 30 is used to acquire the attribute density of each detection data according to the similarity between the attribute information of each detection data and the unstructured data of the same type.

[0118] The importance acquisition module 40 is used to acquire the importance of each detection data according to the uniqueness and attribute density of each detection data.

[0119] The data processing module 50 is used to optimize the PCA dimension reduction algorithm according to the importance level and obtain the detection data after optimization and dimension reduction.

[0120] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the multi-source heterogeneous test detection data analysis and management system provided in the above embodiment and the multi-source heterogeneous test detection data analysis and management method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0121] Embodiment 3:

[0122] The present invention also proposes a multi-source heterogeneous test detection data analysis and management device, which includes a memory and a processor, wherein the memory stores an executable program code, and the processor is used to call and execute the executable program code to perform a multi-source heterogeneous test detection data analysis and management method provided in an embodiment of the present application. The device can be a chip, a component or a module, and the chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a multi-source heterogeneous test detection data analysis and management method provided in the above embodiment.

[0123] In addition, the present application embodiment also protects a computer device, see Figure 4 The computer device includes a memory 401, a processor 402, and a computer program 403 stored in the memory 401 and running on the processor 402, wherein when the processor 402 executes the computer program 403, the computer device can execute any of the multi-source heterogeneous test detection data analysis and management methods introduced above.

[0124] Embodiment 4:

[0125] This embodiment also provides a computer-readable storage medium, which stores computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement a multi-source heterogeneous test detection data analysis and management method provided in the above embodiment.

[0126] Embodiment 5:

[0127] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a multi-source heterogeneous test detection data analysis and management method provided by the above embodiment.

[0128] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0129] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0130] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A method for analyzing and managing multi-source heterogeneous test detection data, characterized in that: The method comprises the following steps: Obtain various detection data of transportation infrastructure within a preset time period; Classify all test data into types, and obtain the uniqueness of each test data according to the change difference and correlation between each test data and other test data of the same type, as well as the distribution of each test data; According to the similarity between the attribute information of each detection data and the unstructured data of the same type, the attribute density of each detection data is obtained; Obtain the importance of each test data according to its uniqueness and attribute density; The PCA dimensionality reduction algorithm is optimized according to the importance level to obtain the detection data after optimized dimensionality reduction.

2. A method for analyzing and managing multi-source heterogeneous test detection data as claimed in claim 1, characterized in that: The method for obtaining the uniqueness is: According to the difference between each test data and other test data of the same type, the sensitivity of each test data is obtained; Obtaining the correlation degree of each test data according to the Pearson correlation coefficient between each test data and each other test data of the same type within a preset time period; According to the distribution of each test data within a preset time period, obtain the degree of change of each test data; The uniqueness of each detection data is obtained according to its sensitivity, correlation and variation; wherein the sensitivity and variation are positively correlated with the uniqueness, and the correlation is negatively correlated with the uniqueness.

3. A method for analyzing and managing multi-source heterogeneous test detection data as described in claim 2, characterized in that: The method for obtaining the sensitivity is: For any type of detection data, the detection data within a preset time period is plotted as a broken line according to the time sequence, and an upper envelope and a lower envelope of the broken line are obtained; Obtaining the absolute values ​​of the tangent slopes of the upper envelope and the lower envelope at each point of the fold line in time sequence, and sequentially constructing an upper envelope slope sequence and a lower envelope slope sequence of the detection data; The average of the absolute values ​​of the tangent slopes in the same order in all upper envelope slope sequences of the type to which the detection data belongs is used as the reference value in the corresponding order of the upper envelope slope sequence of the type to which the detection data belongs; The average of the absolute values ​​of the tangent slopes in the same order in all the lower envelope slope sequences of the type to which the detection data belongs is used as the reference value in the corresponding order of the lower envelope slope sequence of the type to which the detection data belongs; For any order, the difference between the absolute value of the tangent slope of the upper envelope slope sequence of the detection data in the order and the reference value of the upper envelope slope sequence of the type to which the detection data belongs in the order is taken as the discrete degree of the absolute value of the tangent slope of the upper envelope slope sequence of the detection data in the order; The difference between the absolute value of the tangent slope of the lower envelope slope sequence of the detection data in the order and the reference value of the lower envelope slope sequence of the type to which the detection data belongs in the order is used as the discrete degree of the absolute value of the tangent slope of the lower envelope slope sequence of the detection data in the order; The sum of the discrete degrees of the absolute values ​​of the tangent slopes of the upper envelope slope sequence and the lower envelope slope sequence of the detection data in the order is taken as the overall discrete degree of the detection data in the order; The sensitivity of the detection data is obtained according to the overall discreteness of the detection data in each order; wherein the overall discreteness is positively correlated with the sensitivity.

4. A method for analyzing and managing multi-source heterogeneous test detection data as claimed in claim 3, characterized in that: The method for obtaining the degree of change is: For any type of detection data, the reference values ​​of each order of the upper envelope slope sequence in the type to which the detection data belongs are respectively negatively correlated and normalized, and the results are all used as the first correction weight; Correcting the variance and mean of the absolute value of the tangent slope in the upper envelope slope sequence of the detection data by using the first correction weight, respectively, to obtain the corrected variance and corrected mean of the upper envelope slope sequence of the detection data; The reference values ​​of each order of the lower envelope slope sequence of the type to which the detection data belongs are respectively negatively correlated and normalized, and the results are all used as the second correction weight; Correcting the variance and mean of the absolute value of the tangent slope in the lower envelope slope sequence of the detection data by using the second correction weight, respectively, to obtain the corrected variance and corrected mean of the lower envelope slope sequence of the detection data; The mean of the corrected variances of the upper envelope slope sequence and the lower envelope slope sequence of the detection data is taken as the overall fluctuation value; The average of the corrected mean values ​​of the upper envelope slope sequence and the lower envelope slope sequence of the detection data is taken as the overall change rate value; According to the overall fluctuation value and the overall change rate value, the change degree of the detection data is obtained; wherein the overall fluctuation value and the overall change rate value are both positively correlated with the change degree.

5. A method for analyzing and managing multi-source heterogeneous test detection data as claimed in claim 4, characterized in that: The calculation formulas for the corrected variance and corrected mean of the upper envelope slope sequence are: The calculation formula of the corrected variance is: The calculation formula of the modified mean is: In the formula, is the corrected variance of the upper envelope slope sequence A of the k-th detection data; I is the total number of elements in the upper envelope slope sequence A of the k-th detection data; is the reference value of the upper envelope slope sequence A of the i-th order in the type to which the k-th detection data belongs; norm is the normalization function; is the first modified weight; a k,i is the absolute value of the i-th tangent slope in the upper envelope slope sequence A of the k-th detection data; is the mean of the absolute values ​​of all tangent slopes in the upper envelope slope sequence A of the k-th detection data; is the corrected mean of the upper envelope slope sequence A of the k-th detection data.

6. A method for analyzing and managing multi-source heterogeneous test detection data as claimed in claim 1, characterized in that: The method for obtaining the attribute density is: For any type of detection data, obtain the attribute information of the detection data and the word vector of the unstructured data of the same type as the detection data; For any word vector corresponding to the attribute information of the detection data, obtain the mean of the modified cosine similarity between the word vector and each word vector of the unstructured data of the same type as the detection data as the first eigenvalue of the word vector; The average of the first eigenvalues ​​of all word vectors corresponding to the attribute information of the detection data is used as the attribute density value of the detection data.

7. A method for analyzing and managing multi-source heterogeneous test detection data as claimed in claim 1, characterized in that: The method for obtaining the importance is: The result of normalizing the product of the uniqueness value and the attribute density value of each detection data is taken as the importance value of each detection data.

8. A method for analyzing and managing multi-source heterogeneous test detection data as claimed in claim 1, characterized in that: The method of optimizing the PCA dimensionality reduction algorithm according to the importance and obtaining the detection data after the optimization and dimensionality reduction is as follows: For any type, construct the detection data of the type into an initial matrix, wherein one row of the initial matrix corresponds to one type of detection data; Each row of data in the initial matrix is ​​processed by zero mean to obtain a first process matrix; Multiply the importance value of each detection data of the type by each element in the corresponding row of the first process matrix to obtain a second process matrix; Obtaining eigenvalues ​​and eigenvectors of the covariance matrix of the second process matrix, and arranging the eigenvectors from top to bottom in rows according to the order of corresponding eigenvalues ​​from large to small to form a third process matrix; The matrix consisting of the first preset number of rows in the third process matrix is ​​used as a dimension reduction matrix; The initial matrix is ​​multiplied by the dimension reduction matrix to obtain an optimized dimension reduction matrix; the data in the optimized dimension reduction matrix is ​​the detection data after optimized dimension reduction.

9. A method for analyzing and managing multi-source heterogeneous test detection data as claimed in claim 8, characterized in that: The method for obtaining the preset number is: Obtain the cumulative contribution rate of each eigenvalue, and add up the cumulative contribution rates from large to small. When the cumulative result is greater than the preset cumulative contribution rate threshold, stop adding up. The number of eigenvalues ​​corresponding to the accumulation process is used as the preset number.

10. A method for analyzing and managing multi-source heterogeneous test detection data as claimed in claim 1, characterized in that: The method for classifying all detection data into types is as follows: All detection data are divided into types according to different collection methods.

Images