Data processing method and device based on data consistency analysis and electronic equipment

Abstract

The invention provides a data processing method and device based on data consistency analysis and electronic equipment, and relates to the technical field of data processing.The method comprises the steps that firstly, index names of all indexes are extracted from a plurality of structured data tables so as to construct an original index name library; then determining a similar index name set for a target index name based on a text similarity coefficient between the index names, and finally combining the target index and the similar index on the premise that the index unit and the index data magnitude are consistent to obtain an integrated structured data table and a reference index name library. According to the data processing method based on data consistency analysis, a text similarity coefficient, an index unit and an index data magnitude are taken as references, data indexes with the same meaning are combined, the consistency and uniqueness of each index in an integrated structured data table are ensured, and the data consistency analysis is realized. And the correctness and reliability of the data system are maintained.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method, device and electronic equipment based on data consistency analysis. Background Art

[0002] According to the Data Management Knowledge Manual of the International Data Management Association, data quality refers to "both the characteristics related to the data and the processes used to measure or improve the quality of the data." Data quality directly affects the accuracy and reliability of the decision-making of the data user, among which data consistency is an important aspect of data quality management. In data management, it is necessary to ensure the consistency of data to avoid confusion and misunderstanding of data. Data consistency means that the connotation and definition of data should be consistent at different time points, so it is necessary to standardize and unify the data to ensure that the understanding and use of data are carried out under the same standard. The structured data in the field of comprehensive transportation has the characteristics of multiple sources, quantity, integrity, and continuity. The consistency analysis of the data is to understand the correlation and logical relationship between the data in different survey projects, different institutions, and different periods, and to measure the degree of coordination between different statistical data. It is a requirement for data connection and matching.

[0003] However, in the existing technology, there is no efficient, convenient and accurate automated method for consistency analysis and processing of structured data in the field of comprehensive transportation. Manual analysis has problems such as poor accuracy and low efficiency, and cannot meet the actual needs of large-scale data aggregation and updating. Summary of the invention

[0004] The purpose of the present invention is to provide a data processing method, device and electronic device based on data consistency analysis, so as to alleviate the technical problem in the prior art that it is impossible to efficiently and accurately perform consistency analysis and processing on structured data in the field of comprehensive transportation.

[0005] In a first aspect, the present invention provides a data processing method based on data consistency analysis, comprising: obtaining multiple structured data tables to be processed, and extracting the indicator names of all indicators in the multiple structured data tables to obtain an original indicator name library; calculating the text similarity coefficients between the indicator names in the original indicator name library to obtain a text similarity coefficient set; determining a similar indicator name set of a target indicator name according to the text similarity coefficient set; wherein the target indicator name represents any indicator name in the original indicator name library; the text similarity coefficient between the indicator name in the similar indicator name set and the target indicator name is greater than a preset threshold; when it is determined that the indicator unit and indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name, merging the target similar indicator and the target indicator to obtain an integrated structured data table and a benchmark indicator name library; wherein the target similar indicator name represents any indicator name in the similar indicator name set.

[0006] In an optional implementation, the indicator names of all indicators in multiple structured data tables are extracted to obtain an original indicator name library, including: obtaining the hierarchy of target indicators in multiple structured data tables; wherein the target indicator represents any indicator among all indicators; if the hierarchy of the target indicator is 1, storing the indicator name of the target indicator in the original indicator name library; if the hierarchy of the target indicator is greater than 1, performing hierarchy merging processing on the target indicator, and storing the indicator name after the hierarchy merging in the original indicator name library.

[0007] In an optional embodiment, calculating the text similarity coefficient between the indicator names in the original indicator name library includes: identifying and deleting stop words of the first indicator name and the second indicator name in the original indicator name library to obtain the first valid indicator name and the second valid indicator name; wherein the first indicator name and the second indicator name represent any two indicator names in the original indicator name library; stringifying the first valid indicator name and the second valid indicator name to obtain a first string and a second string; based on the first string and the second string, determining the number of matching characters and the number of union characters; wherein the number of matching characters represents the number of characters in the first string and the second string that have the same characters and order; the number of union characters represents the number of characters after the characters in the first string and the second string are taken as the union; based on the number of matching characters, the number of union characters, the string length of the first string and the string length of the second string, calculating the text similarity coefficient of the first indicator name and the second indicator name.

[0008] In an optional embodiment, the text similarity coefficient of the first indicator name and the second indicator name is calculated based on the number of matching characters, the number of union characters, the string length of the first string and the string length of the second string, including: determining the maximum value of the string length and the minimum value of the string length based on the string length of the first string and the string length of the second string; calculating the relative length of the first string and the second string based on the maximum value of the string length and the minimum value of the string length; calculating the text similarity coefficient of the first indicator name and the second indicator name based on the number of matching characters, the number of union characters and the relative length.

[0009] In an optional implementation, the target similarity indicator and the target indicator are merged, including: determining whether there is compensation data in the indicator data of the target similarity indicator; wherein the compensation data represents the indicator data missing from the target indicator; if so, merging the compensation data into the indicator data of the target indicator and deleting the target similarity indicator; if not, deleting the target similarity indicator.

[0010] In an optional implementation, it also includes: obtaining the measurement unit corresponding to the target similarity indicator name and the measurement unit corresponding to the target indicator name; judging whether there is a multiple relationship between the measurement unit corresponding to the target similarity indicator name and the measurement unit corresponding to the target indicator name; if so, obtaining the indicator data of the target similarity indicator and the indicator data of the target indicator, and unifying the units of the indicator data to obtain first indicator data and second indicator data; wherein, the first indicator data represents the data after the indicator data of the target similarity indicator is unified in units; the second indicator data represents the data after the indicator data of the target indicator is unified in units; judging whether the difference between the first indicator data and the second indicator data is less than one data order of magnitude; if so, determining that the indicator unit and the indicator data order of magnitude corresponding to the target similarity indicator name are the same as the indicator unit and the indicator data order of magnitude corresponding to the target indicator name.

[0011] In an optional implementation, based on the number of matching characters, the number of union characters and the relative length, calculating the text similarity coefficient of the first indicator name and the second indicator name includes: using the formula Calculate the text similarity coefficient between the first indicator name and the second indicator name; where m represents

[0012] Indicates the number of matching characters, s indicates the number of combined characters, represents the relative length, s1 represents the string length of the first string, s2 represents the string length of the second string, and d represents the text similarity coefficient between the first indicator name and the second indicator name.

[0013] In a second aspect, the present invention provides a data processing device based on data consistency analysis, comprising: an acquisition module, used to acquire multiple structured data tables to be processed, and extract the indicator names of all indicators in the multiple structured data tables to obtain an original indicator name library; a calculation module, used to calculate the text similarity coefficients between the indicator names in the original indicator name library to obtain a text similarity coefficient set; a determination module, used to determine a similar indicator name set of a target indicator name according to the text similarity coefficient set; wherein the target indicator name represents any indicator name in the original indicator name library; the text similarity coefficient between the indicator name in the similar indicator name set and the target indicator name is greater than a preset threshold; a merging module, used to merge the target similar indicator and the target indicator when it is determined that the indicator unit and indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name, to obtain an integrated structured data table and a benchmark indicator name library; wherein the target similar indicator name represents any indicator name in the similar indicator name set.

[0014] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the steps of the data processing method based on data consistency analysis in any one of the aforementioned embodiments are implemented.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the data processing method based on data consistency analysis of any one of the aforementioned implementation modes is implemented.

[0016] The present invention provides a data processing method based on data consistency analysis, which first extracts the indicator names of all indicators from multiple structured data tables to construct an original indicator name library, then determines a set of similar indicator names for a target indicator name based on the text similarity coefficient between the indicator names, and finally merges the target indicator and similar indicators on the premise that the indicator unit and the indicator data magnitude are consistent, to obtain an integrated structured data table and a benchmark indicator name library. This method proposes a data processing method based on data consistency analysis in a breakthrough way, which merges data indicators with the same meaning with reference to the text similarity coefficient, the indicator unit and the indicator data magnitude, and ensures the consistency and uniqueness of each indicator in the integrated structured data table, which can provide support for data analysis of structured data in the field of comprehensive transportation, prevent data confusion and wrong decisions, and maintain the correctness and reliability of the data system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a data processing method based on data consistency analysis provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a principle for determining the number of matching characters provided by an embodiment of the present invention;

[0020] Figure 3 A functional module diagram of a data processing device based on data consistency analysis provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] The consistency of basic structured data in the field of comprehensive transportation mainly includes the uniformity of naming, that is, ensuring that the same type of data uses the same naming rules in different data sources and at different time points; the uniformity of data structure, ensuring that the data model and architecture remain consistent in different data sources and at different time points. At present, there is no efficient, convenient, and accurate automated method for consistency analysis of structured data in this field. Manual analysis has problems such as poor accuracy and low efficiency, and cannot meet the actual needs of large-scale data aggregation and updating. In view of this, an embodiment of the present invention provides a data processing method based on data consistency analysis to alleviate the technical problems involved above.

[0026] Embodiment 1

[0027] Figure 1 A flowchart of a data processing method based on data consistency analysis provided by an embodiment of the present invention, such as Figure 1 As shown, the method specifically comprises the following steps:

[0028] Step S102, obtaining multiple structured data tables to be processed, and extracting the indicator names of all indicators in the multiple structured data tables to obtain an original indicator name library.

[0029] The method provided in the embodiment of the present invention can be used to analyze and process basic structured data in various statistical yearbooks in the field of comprehensive transportation, such as the basic structured data in mainstream statistical yearbooks in the field of comprehensive transportation, such as the "Highway Maintenance Yearbook", "China Highway Network Operation Blue Book", and "International Statistical Yearbook". Generally, the data processing object of the embodiment of the present invention is multiple structured data tables obtained from one data interface, and one data interface can be understood as the same data source.

[0030] Due to differences in data statisticians / data statistics time, data indicators (referred to as indicators) with the same physical meaning may use different indicator names. Therefore, after obtaining multiple structured data tables, the indicator names of all indicators are first extracted from the multiple structured data tables to construct an original indicator name library, and then the above multiple structured data tables are integrated through consistency analysis of the indicator names in the original indicator name library.

[0031] Step S104, calculating the text similarity coefficients between the indicator names in the original indicator name library to obtain a text similarity coefficient set.

[0032] The text similarity coefficient between indicator names can characterize the similarity and difference between indicator names. Therefore, the embodiment of the present invention measures the consistency of indicator names by determining the text similarity coefficient between indicator names. If the text similarity coefficient between indicator name A and indicator name B is very high, it is considered that the consistency between the two is good. Therefore, after determining the original indicator name library, the embodiment of the present invention further calculates the text similarity coefficient between any two indicator names in the original indicator name library. After all traversals are completed, a text similarity coefficient set can be obtained, that is, a set composed of the calculation results of the text similarity coefficients between all indicator names.

[0033] Step S106: determining a similar indicator name set of the target indicator name according to the text similarity coefficient set.

[0034] The target indicator name represents any indicator name in the original indicator name library; the text similarity coefficient between the indicator names in the similar indicator name set and the target indicator name is greater than a preset threshold.

[0035] After obtaining the set of text similarity coefficients by calculation, taking the target indicator name as an example, if the text similarity coefficient between a certain indicator name in the original indicator name library and the target indicator name is greater than a preset threshold, then the indicator name can be called a similar indicator name of the target indicator name. Therefore, by traversing the calculation results of the text similarity coefficients between the target indicator name and other indicator names, and screening using the preset threshold, a set of similar indicator names of the target indicator name can be determined. The embodiment of the present invention does not specifically limit the value of the preset threshold, and the user can choose according to the actual situation. Optionally, the preset threshold is 50%.

[0036] Step S108, when it is determined that the indicator unit and indicator data magnitude corresponding to the target similarity indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name, merge the target similarity indicator and the target indicator to obtain an integrated structured data table and a benchmark indicator name library.

[0037] The target similarity indicator name represents any indicator name in the similarity indicator name set.

[0038] Specifically, in order to ensure the accuracy of indicator merging, the embodiment of the present invention does not directly merge the indicator data of different indicators whose text similarity coefficients are greater than a preset threshold, but needs to further consider the indicator units and indicator data magnitudes of the target indicator and the target similarity indicator. If the indicator units and / or indicator data magnitudes of the target indicator and the target similarity indicator are different, then it is determined that the target indicator and the target similarity indicator are different indicators, that is, the two cannot be merged.

[0039] If the target indicator and the target similar indicator have the same indicator unit and indicator data magnitude, then it is determined that the target indicator and the target similar indicator are the same indicator. In this case, the target similar indicator and the target indicator need to be merged to ensure the uniqueness of the physical meaning of each indicator. The merging of indicators includes: merging of indicator names and merging of indicator data. The merging of indicator names can be to randomly select one from the two merged indicator names to be retained, or to select an indicator name with more or fewer characters according to needs; the merging of indicator data is to take the union of indicator data. After all similar indicators are merged, the integrated structured data table and the benchmark indicator name library are obtained. According to the above introduction, the benchmark indicator name library is the result of deleting some indicator names from the original indicator name library, and the indicator represented by each indicator name in the benchmark indicator name library is unique.

[0040] The embodiment of the present invention provides a data processing method based on data consistency analysis, which first extracts the indicator names of all indicators from multiple structured data tables to construct an original indicator name library, then determines a set of similar indicator names for the target indicator name based on the text similarity coefficient between the indicator names, and finally merges the target indicator and the similar indicator under the premise that the indicator unit and the indicator data magnitude are consistent, to obtain an integrated structured data table and a benchmark indicator name library. This method has proposed a breakthrough data processing method based on data consistency analysis, which uses the text similarity coefficient, indicator unit and indicator data magnitude as references to merge data indicators with the same meaning, ensures the consistency and uniqueness of each indicator in the integrated structured data table, and can provide support for data analysis of structured data in the field of comprehensive transportation, prevent data confusion and wrong decisions, and maintain the correctness and reliability of the data system.

[0041] In an optional implementation, in the above step S102, the indicator names of all indicators in the multiple structured data tables are extracted to obtain an original indicator name library, which specifically includes the following contents:

[0042] Get the hierarchy of target indicators in multiple structured data tables; where the target indicator represents any indicator among all indicators.

[0043] If the level of the target indicator is 1, the indicator name of the target indicator is stored in the original indicator name library; if the level of the target indicator is greater than 1, the target indicator is hierarchically merged, and the indicator name after hierarchical merging is stored in the original indicator name library.

[0044] For ease of understanding, the extraction of indicator names is illustrated below. Table 1 is the indicator name portion of a structured data table. Table 1 includes both level 2 indicators and level 1 indicators. The level of the indicator "cameras along road sections and bridges" is 1, so its indicator name "cameras along road sections and bridges" is stored in the original indicator name library; the level of the indicator "cumulative since the beginning of the year" is 2, so the indicator needs to be hierarchically merged. If different levels are connected by ":", the name of the indicator after the hierarchical merger is "highway passenger volume: cumulative since the beginning of the year", that is, it is arranged in order from the top to the bottom. The embodiment of the present invention does not specifically limit the connection symbols between levels when merging levels, and it is also possible to choose not to use any symbol connection.

[0045] It should be noted that if each indicator in the structured data table is also marked with the corresponding unit of measurement, for example, year-on-year growth rate (%), cameras along road sections and bridges (number), then the starting character "(" of the brackets can be set as a stop extraction mark, because the brackets and the characters inside the brackets are not indicator names.

[0046] Table 1 Index names of structured data tables

[0047]

[0048] In an optional implementation, the above step S104, calculating the text similarity coefficients between the indicator names in the original indicator name library, specifically includes the following steps:

[0049] Step S1041, identifying and deleting stop words of the first indicator name and the second indicator name in the original indicator name library, to obtain the first valid indicator name and the second valid indicator name; wherein the first indicator name and the second indicator name represent any two indicator names in the original indicator name library.

[0050] Step S1042, performing string processing on the first valid indicator name and the second valid indicator name to obtain a first string and a second string.

[0051] Step S1043, based on the first character string and the second character string, determine the number of matching characters and the number of union characters; wherein the number of matching characters refers to the number of characters in the first character string and the second character string that have the same characters and order; the number of union characters refers to the number of characters after the characters in the first character string and the second character string are taken as the union.

[0052] Step S1044, calculating the text similarity coefficient between the first indicator name and the second indicator name based on the number of matching characters, the number of union characters, the string length of the first string and the string length of the second string.

[0053] Specifically, to calculate the text similarity between index names, the embodiments of the present invention need to stringify all index names in the original index name library (only retain the text), and before the stringification process, stop word filtering needs to be performed first to remove words that have no actual meaning or are insignificant in natural language processing, such as "of", "is", "have", etc., so as to improve the accuracy and efficiency when calculating the text similarity coefficient later. The embodiments of the present invention do not specifically limit the order of stop word filtering and stringification operations, and it is also possible to perform stringification first and then stop word filtering. Table 2 below exemplarily shows the results of stringifying some index names.

[0054] Table 2 Index names and corresponding strings

[0055]

[0056] To calculate the text similarity coefficient between the first index name and the second index name in the original index name library, the number of matching characters and the number of union characters of the two should be determined after removing stop words and stringifying the first index name and the second index name. Figure 2 This is a schematic diagram of the principle for determining the number of matching characters provided by the embodiments of the present invention. Taking the first string as {road, section, bridge, beam, along, line, camera} and the second string as {road, section, bridge, beam, along, line, camera, quantity} as an example, according to the characters and the order of the characters in the strings, the characters with the same characters and order in the first string and the second string are {road, section, bridge, beam, along, line, camera}, that is, the number of matching characters is 9. And, the character union of the first string and the second string is {road, section, bridge, beam, along, line, camera, quantity}, that is, the number of union characters is 11.

[0057] After determining the number of matching characters and the number of union characters of the first string and the second string, combined with the string length of the first string and the string length of the second string, the text similarity coefficient between the first index name and the second index name can be calculated.

[0058] In an alternative embodiment, in step S1044 above, calculating the text similarity coefficient between the first index name and the second index name based on the number of matching characters, the number of union characters, the string length of the first string, and the string length of the second string specifically includes the following steps:

[0059] Step S10441, based on the string length of the first string and the string length of the second string, determine the maximum value of the string length and the minimum value of the string length.

[0060] Step S10442: Calculate the relative lengths of the first character string and the second character string based on the maximum character string length and the minimum character string length.

[0061] Step S10443, based on the number of matching characters, the number of union characters and the relative length, calculate the text similarity coefficient of the first indicator name and the second indicator name.

[0062] In an embodiment of the present invention, the text similarity coefficient between the indicator names is calculated based on the ratio of the number of matching characters to the number of union characters, while taking into account the relative lengths of the two character strings. Optionally, based on the number of matching characters, the number of union characters and the relative length, the text similarity coefficient between the first indicator name and the second indicator name is calculated, including:

[0063] Using the formula Calculate the text similarity coefficient between the first indicator name and the second indicator name; where m represents the number of matching characters, and s represents the number of union characters. represents the relative length, s1 represents the string length of the first string, s2 represents the string length of the second string, and d represents the text similarity coefficient between the first indicator name and the second indicator name.

[0064] For example, the first indicator name "Cameras along road sections and bridges" corresponds to the first character string r1 = (road, section, bridge, beam, along, line, camera, image, machine), and the character string length of the first character string s1 = 9; the second indicator name "Cameras along road sections and bridges: quantity" corresponds to the second character string r2 = (road, section, bridge, beam, along, line, camera, image, machine, number, quantity), and the character string length of the second character string s2 = 11. By comparing r1 and r2, the number of matching characters m = 9. The number of union characters s = 11. Comparing the character string lengths, max(s1, s2) is 11, and min(s1, s2) is 9. Therefore, the text similarity coefficient d between the first indicator name and the second indicator name is 66.94%. The following Table 3 is a schematic table of the calculation results of the text similarity coefficients between indicator names.

[0065] Table 3 Calculation results of text similarity coefficients between indicator names

[0066]

[0067]

[0068] If the threshold of the text similarity coefficient of the basic structured data indicator name in the comprehensive transportation field is set at 50%, that is, if it is greater than or equal to 50%, it means that the two indicator names have a high similarity. For example, in Table 3, "Cameras along road sections and bridges" and "Cameras along road sections and bridges: quantity" have a high similarity, and "Total business volume of postal industry: express business: international" and "Postal industry delivery business volume: express business: international" have a high similarity.

[0069] In an optional implementation, in the above step S108, merging the target similarity index and the target index specifically includes the following steps:

[0070] Step S1081, determining whether there is compensation data in the indicator data of the target similar indicator; wherein the compensation data represents the indicator data missing from the target indicator.

[0071] If it exists, execute the following step S1082; if it does not exist, execute the following step S1083.

[0072] Step S1082, merging the compensation data into the indicator data of the target indicator, and deleting the target similarity indicator.

[0073] Step S1083, delete the target similarity index.

[0074] After determining that the indicator unit and indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name, the target similar indicator and the target indicator can be merged. Specifically, the indicator data of the target indicator is used as a reference to determine whether there is indicator data (that is, compensation data) in the indicator data of the target similar indicator that is not in the target indicator. If so, the compensation data is merged into the indicator data of the target indicator. After the indicator data is merged, there is no need to retain the target similar indicator. If there is no compensation data in the indicator data of the target similar indicator, it means that the indicator data of the target indicator already contains the indicator data of the target similar indicator. Therefore, in order to ensure the uniqueness of the indicator, the target similar indicator is deleted.

[0075] For example, the statistical years corresponding to the indicator data of the target indicator are 2000-2003, and the statistical years corresponding to the indicator data of the target similar indicator are 2004-2006. Obviously, there are compensation data in the indicator data of the target similar indicator. The compensation data is merged into the indicator data of the target indicator, and the statistical years corresponding to the indicator data of the target indicator after the update are 2000-2006.

[0076] In an optional implementation, in order to determine whether the indicator unit and indicator data magnitude corresponding to the target similarity indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name, the embodiment of the present invention further includes the following content:

[0077] Get the measurement units corresponding to the target similarity indicator name and the measurement units corresponding to the target indicator name.

[0078] Determine whether there is a multiple relationship between the measurement unit corresponding to the target similarity indicator name and the measurement unit corresponding to the target indicator name.

[0079] If it exists, obtain the indicator data of the target similar indicator and the indicator data of the target indicator, and unify the units of the indicator data to obtain the first indicator data and the second indicator data; wherein the first indicator data represents the data after the indicator data of the target similar indicator is unified in units; the second indicator data represents the data after the indicator data of the target indicator is unified in units.

[0080] Determine whether the difference between the first indicator data and the second indicator data is less than one data order of magnitude.

[0081] If so, it is determined that the indicator unit and indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name.

[0082] Specifically, taking the target indicator name as "cameras along road sections and bridges" and the target similarity indicator name as "number of cameras along road sections and bridges" as an example, by querying the structured data table, it can be determined that the unit of measurement of "cameras along road sections and bridges" is "set" and the unit of measurement of "number of cameras along road sections and bridges" is "piece". In view that there is no multiple relationship between the measurement units of the two, we give up merging the indicator data of the above two indicators.

[0083] Taking the target indicator named "Total business volume of postal industry: express business: international" and the target similarity indicator named "Postal industry delivery business volume: express business: international" as an example, by querying the structured data table, it is determined that the measurement units of the two are "10,000 pieces", that is, there is a multiple relationship between the measurement units (the multiple is 1), then continue to execute the judgment whether the difference between the indicator data is less than one data magnitude.

[0084] Data magnitude refers to the scale or level of the quantity. A fixed ratio is maintained between adjacent data magnitudes. The commonly used ratios are 10, 2, 1000, 1024, and e (that is, the base of the natural logarithm). According to the study of data samples in the field of comprehensive transportation, the ratio between adjacent data magnitudes can be selected as 10 for the indicator data in the field of comprehensive transportation. In other words, if the first indicator data and the second indicator data are divided, the result is between 0.1 (the inverse of the ratio between data magnitudes) and 10 (the ratio between data magnitudes), that is, it is determined that the difference between the two is less than one data magnitude. For example, 99 divided by 100 is 0.99, and 0.99 is in the range of 0.1 to 10, that is, the difference between the two is less than one data magnitude; 1100 divided by 100 is 11, and 11 exceeds the range of 0.1 to 10, that is, it is determined that the difference between the two is greater than one data magnitude.

[0085] If the multiple is 1, the indicator data remains unchanged before and after the unit is unified, and it is determined whether the difference between the first indicator data and the second indicator data is less than one data magnitude. If the difference is less than one data magnitude, it is determined that the indicator unit and the indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and the indicator data magnitude corresponding to the target indicator name. In the embodiment of the present invention, the indicator unit refers to a measurement unit without a data magnitude, for example, the measurement unit "10,000 pieces", where the data magnitude is "10,000", after removing the data magnitude, the corresponding indicator unit is "piece".

[0086] Continuing with the target indicator named "Total business volume of postal industry: express delivery business: international" and the target similarity indicator named "Postal industry mailing business volume: express delivery business: international" as an example, if we query the structured data table, we find that the unit of measurement for "Total business volume of postal industry: express delivery business: international" is "pieces", and the unit of measurement for "Postal industry mailing business volume: express delivery business: international" is "ten thousand pieces". It can be seen that there is a multiple relationship between the units of measurement, and the multiple is ten thousand.

[0087] When the multiple is greater than 1, it is necessary to unify the units of the indicator data of the target similar indicator and the indicator data of the target indicator before judging whether the difference between the two is less than one data magnitude. For example, the indicator data of "postal industry business volume: express business: international" is 10,000, and the unit of measurement is piece, while the indicator data of "postal industry delivery business volume: express business: international" is 2, and the unit of measurement is ten thousand pieces. If the unit is unified as piece, the indicator data after unification are 10,000 and 20,000 respectively. By comparing the ratio 0.5 with the ratio between the preset data magnitudes (that is, 10), since 0.5 is within the interval (0.1, 10), it is determined that the difference between the two is less than one data magnitude, and it is determined that the indicator unit and indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name.

[0088] In summary, the embodiments of the present invention provide a complete data processing method based on data consistency analysis in a breakthrough manner, which can calculate the similarity of indicator names between various data in the comprehensive transportation field, and make data standardization judgments on this basis, merge data indicators with the same meaning and the same statistical approach, ensure the consistency and uniqueness of each indicator, and provide support for the subsequent analysis and use of transportation industry data, prevent data confusion and wrong decisions, and maintain the correctness and reliability of the data system.

[0089] Embodiment 2

[0090] An embodiment of the present invention also provides a data processing device based on data consistency analysis, which is mainly used to execute the data processing method based on data consistency analysis provided in the above-mentioned embodiment 1. The data processing device based on data consistency analysis provided in the embodiment of the present invention is specifically introduced below.

[0091] Figure 3 A functional module diagram of a data processing device based on data consistency analysis provided by an embodiment of the present invention, such as Figure 3 As shown, the device mainly includes: an acquisition module 10, a calculation module 20, a determination module 30, and a merging module 40, wherein:

[0092] The acquisition module 10 is used to acquire multiple structured data tables to be processed, and extract the indicator names of all indicators in the multiple structured data tables to obtain an original indicator name library.

[0093] The calculation module 20 is used to calculate the text similarity coefficients between the indicator names in the original indicator name library to obtain a text similarity coefficient set.

[0094] The determination module 30 is used to determine a set of similar indicator names of the target indicator name based on a set of text similarity coefficients; wherein the target indicator name represents any indicator name in the original indicator name library; and the text similarity coefficient between the indicator names in the similar indicator name set and the target indicator name is greater than a preset threshold.

[0095] The merging module 40 is used to merge the target similar indicator and the target indicator when it is determined that the indicator unit and the indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and the indicator data magnitude corresponding to the target indicator name, so as to obtain an integrated structured data table and a benchmark indicator name library; wherein the target similar indicator name represents any indicator name in the similar indicator name set.

[0096] The embodiment of the present invention provides a data processing device based on data consistency analysis, which first extracts the indicator names of all indicators from multiple structured data tables to construct an original indicator name library, then determines a set of similar indicator names for the target indicator name based on the text similarity coefficient between the indicator names, and finally merges the target indicator and the similar indicator under the premise that the indicator unit and the indicator data magnitude are consistent, to obtain an integrated structured data table and a benchmark indicator name library. The device has proposed a breakthrough data processing flow based on data consistency analysis, which merges data indicators with the same meaning with reference to the text similarity coefficient, indicator unit and indicator data magnitude, and ensures the consistency and uniqueness of each indicator in the integrated structured data table, which can provide support for data analysis of structured data in the field of comprehensive transportation, prevent data confusion and wrong decisions, and maintain the correctness and reliability of the data system.

[0097] Optionally, the acquisition module 10 is specifically used for:

[0098] Get the hierarchy of target indicators in multiple structured data tables; where the target indicator represents any indicator among all indicators.

[0099] If the level of the target indicator is 1, the indicator name of the target indicator is stored in the original indicator name library.

[0100] If the level of the target indicator is greater than 1, the target indicator is level-merged and the indicator name after level-merging is stored in the original indicator name library.

[0101] Optionally, the calculation module 20 includes:

[0102] The identification and deletion unit is used to identify and delete stop words of the first indicator name and the second indicator name in the original indicator name library to obtain the first valid indicator name and the second valid indicator name; wherein the first indicator name and the second indicator name represent any two indicator names in the original indicator name library.

[0103] The processing unit is used to perform string processing on the first valid indicator name and the second valid indicator name to obtain a first string and a second string.

[0104] A determination unit is used to determine the number of matching characters and the number of union characters based on the first character string and the second character string; wherein the number of matching characters refers to the number of characters in the first character string and the second character string that have the same characters and order; and the number of union characters refers to the number of characters after the characters in the first character string and the second character string are taken as the union.

[0105] The calculation unit is used to calculate the text similarity coefficient of the first indicator name and the second indicator name based on the number of matching characters, the number of union characters, the string length of the first string and the string length of the second string.

[0106] Optionally, the computing unit includes:

[0107] The determining subunit is used to determine a maximum value of the string length and a minimum value of the string length based on the string length of the first string and the string length of the second string.

[0108] The first calculation subunit is used to calculate the relative length of the first character string and the second character string based on the maximum character string length and the minimum character string length.

[0109] The second calculation subunit is used to calculate the text similarity coefficient of the first indicator name and the second indicator name based on the number of matching characters, the number of union characters and the relative length.

[0110] Optionally, the merging module 40 is specifically used for:

[0111] Determine whether there is compensation data in the indicator data of the target similar indicator; wherein the compensation data represents the indicator data missing in the target indicator.

[0112] If it exists, the compensation data is merged into the indicator data of the target indicator, and the target similar indicator is deleted.

[0113] If not present, remove the target similarity indicator.

[0114] Optionally, the device is also used for:

[0115] Get the measurement units corresponding to the target similarity indicator name and the measurement units corresponding to the target indicator name.

[0116] Determine whether there is a multiple relationship between the measurement unit corresponding to the target similarity indicator name and the measurement unit corresponding to the target indicator name.

[0117] If it exists, obtain the indicator data of the target similar indicator and the indicator data of the target indicator, and unify the units of the indicator data to obtain the first indicator data and the second indicator data; wherein the first indicator data represents the data after the indicator data of the target similar indicator is unified in units; the second indicator data represents the data after the indicator data of the target indicator is unified in units.

[0118] Determine whether the difference between the first indicator data and the second indicator data is less than one data order of magnitude.

[0119] If yes, it is determined that the indicator unit and indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name.

[0120] Optionally, the second computing subunit is specifically configured to:

[0121] Using the formula Calculate the text similarity coefficient between the first indicator name and the second indicator name; where m represents the number of matching characters, and s represents the number of union characters. represents the relative length, s1 represents the string length of the first string, s2 represents the string length of the second string, and d represents the text similarity coefficient between the first indicator name and the second indicator name.

[0122] Embodiment 3

[0123] See also Figure 4 An embodiment of the present invention provides an electronic device, which includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0124] The memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0125] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0126] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0127] The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 60. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

[0128] A computer program product of a data processing method, device and electronic device based on data consistency analysis provided in an embodiment of the present invention includes a computer-readable storage medium storing non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0129] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0131] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0132] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0133] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0134] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on data consistency analysis, It is characterized in that include: Acquire multiple structured data tables to be processed, and extract indicator names of all indicators in the multiple structured data tables to obtain an original indicator name library; Calculating text similarity coefficients between indicator names in the original indicator name library to obtain a text similarity coefficient set; Determine a set of similar indicator names of the target indicator name according to the text similarity coefficient set; wherein the target indicator name represents any indicator name in the original indicator name library; and the text similarity coefficient between the indicator names in the similar indicator name set and the target indicator name is greater than a preset threshold; When it is determined that the indicator unit and indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name, the target similar indicator and the target indicator are merged to obtain an integrated structured data table and a benchmark indicator name library; wherein the target similar indicator name represents any indicator name in the similar indicator name set.

2. The data processing method based on data consistency analysis according to claim 1, It is characterized in that Extract the indicator names of all indicators in the plurality of structured data tables to obtain an original indicator name library, including: Obtaining the hierarchy of target indicators in the plurality of structured data tables; wherein the target indicator represents any indicator among all the indicators; If the level of the target indicator is 1, the indicator name of the target indicator is stored in the original indicator name library; If the level of the target indicator is greater than 1, the target indicator is subjected to level merging processing, and the indicator name after level merging is stored in the original indicator name library.

3. The data processing method based on data consistency analysis according to claim 1, It is characterized in that Calculating the text similarity coefficients between the indicator names in the original indicator name library includes: Identify and delete stop words of the first indicator name and the second indicator name in the original indicator name library to obtain a first valid indicator name and a second valid indicator name; wherein the first indicator name and the second indicator name represent any two indicator names in the original indicator name library; Performing string processing on the first valid indicator name and the second valid indicator name to obtain a first string and a second string; Based on the first character string and the second character string, determine the number of matching characters and the number of union characters; wherein the number of matching characters refers to the number of characters in the first character string and the second character string that have the same characters and the same order; the number of union characters refers to the number of characters after the characters in the first character string and the second character string are taken as the union; Based on the number of matching characters, the number of union characters, the string length of the first string and the string length of the second string, a text similarity coefficient of the first indicator name and the second indicator name is calculated.

4. The data processing method based on data consistency analysis according to claim 3, It is characterized in that Calculating the text similarity coefficient of the first indicator name and the second indicator name based on the number of matching characters, the number of union characters, the string length of the first string, and the string length of the second string, including: Based on the string length of the first string and the string length of the second string, determining a maximum value of the string length and a minimum value of the string length; Calculating the relative lengths of the first character string and the second character string based on the maximum value of the character string length and the minimum value of the character string length; Based on the number of matching characters, the number of union characters and the relative length, a text similarity coefficient between the first indicator name and the second indicator name is calculated.

5. The data processing method based on data consistency analysis according to claim 1, It is characterized in that Combine target similarity indicators with target indicators, including: Determine whether there is compensation data in the indicator data of the target similar indicator; wherein the compensation data represents the indicator data missing in the target indicator; If it exists, the compensation data is merged into the indicator data of the target indicator, and the target similarity indicator is deleted; If it does not exist, delete the target similarity indicator.

6. The data processing method based on data consistency analysis according to claim 1, It is characterized in that Also includes: Obtain the measurement unit corresponding to the target similarity indicator name and the measurement unit corresponding to the target indicator name; Determine whether there is a multiple relationship between the measurement unit corresponding to the target similarity indicator name and the measurement unit corresponding to the target indicator name; If so, the indicator data of the target similarity indicator and the indicator data of the target indicator are obtained, and the indicator data are unified in units to obtain first indicator data and second indicator data; wherein the first indicator data represents the indicator data of the target similarity indicator after the unit is unified; The second indicator data represents the indicator data of the target indicator after the unit is unified; Determine whether the difference between the first indicator data and the second indicator data is less than one data magnitude; If so, it is determined that the indicator unit and indicator data magnitude corresponding to the target similarity indicator name are the same as the indicator unit and indicator data magnitude corresponding to the target indicator name.

7. The data processing method based on data consistency analysis according to claim 4, It is characterized in that Calculating the text similarity coefficient of the first indicator name and the second indicator name based on the number of matching characters, the number of union characters, and the relative length includes: Using the formula Calculate the text similarity coefficient between the first indicator name and the second indicator name; wherein m represents the number of matching characters, s represents the number of union characters, represents the relative length, s 1 Indicates the length of the first string, s 2 represents the string length of the second string, and d represents the text similarity coefficient between the first indicator name and the second indicator name.

8. A data processing device based on data consistency analysis, It is characterized in that include: An acquisition module, used for acquiring a plurality of structured data tables to be processed, and extracting the indicator names of all indicators in the plurality of structured data tables to obtain an original indicator name library; A calculation module, used for calculating the text similarity coefficients between the indicator names in the original indicator name library to obtain a text similarity coefficient set; A determination module, configured to determine a set of similar indicator names of a target indicator name according to the set of text similarity coefficients; wherein the target indicator name represents any indicator name in the original indicator name library; and the text similarity coefficient between the indicator names in the set of similar indicator names and the target indicator name is greater than a preset threshold; A merging module is used to merge the target similar indicator and the target indicator when it is determined that the indicator unit and the indicator data magnitude corresponding to the target similar indicator name are the same as the indicator unit and the indicator data magnitude corresponding to the target indicator name, so as to obtain an integrated structured data table and a benchmark indicator name library; wherein the target similar indicator name represents any indicator name in the similar indicator name set.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor. It is characterized in that When the processor executes the computer program, the steps of the data processing method based on data consistency analysis described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the data processing method based on data consistency analysis according to any one of claims 1 to 7 is implemented.

Images