Data processing method based on data consistency analysis

Abstract

The invention provides a data processing method based on data consistency analysis, and relates to the technical field of data processing.The method comprises the steps that firstly, index names of all indexes are extracted from a plurality of structured data tables so as to construct an original index name library; then determining a similar index name set for a target index name based on a text similarity coefficient between the index names, and finally combining the target index and the similar index on the premise that the index unit and the index data magnitude are consistent to obtain an integrated structured data table and a reference index name library. According to the data processing method based on data consistency analysis, a text similarity coefficient, an index unit and an index data magnitude are taken as references, data indexes with the same meaning are combined, the consistency and uniqueness of each index in an integrated structured data table are ensured, and the data consistency analysis is realized. And the correctness and reliability of the data system are maintained.

Description

[0001] The present application is a divisional application of the patent application entitled "Data processing method and device based on data consistency analysis and electronic equipment", the original application date is August 22, 2024, the application number is 202411158434.4. TECHNICAL FIELD

[0002] The present application relates to the technical field of data processing, in particular to a data processing method based on data consistency analysis. BACKGROUND

[0003] According to the Data Management Handbook of the International Data Management Association, data quality refers to both the characteristics related to data and the processes used to measure or improve data quality. Data quality directly affects the accuracy and reliability of decision-making by data users. Data consistency is an important aspect of data quality management. In data management, it is necessary to ensure the consistency of data to avoid confusion and misunderstanding of data. The consistency of data means that the connotation and definition of data at different time points should be consistent, so it is necessary to standardize and uniformly regulate data to ensure that the understanding and use of data are carried out under the same standard. The structured data in the field of comprehensive transportation has the characteristics of multiple sources, quantity, integrity and continuity. The consistency analysis of the structured data in the field of comprehensive transportation is to understand the correlation and logical relationship between data in different survey projects, different institutions and different periods, to measure the coordination degree between different statistical data, and to meet the requirements of data connection and matching.

[0004] However, there is no efficient, convenient and accurate automatic method for consistency analysis and processing of structured data in the field of comprehensive transportation in the prior art. Manual analysis has the problems of poor accuracy and low efficiency, and cannot meet the actual demand of large amount of data aggregation and updating. SUMMARY

[0005] The present application aims to provide a data processing method based on data consistency analysis to solve the technical problem that the structured data in the field of comprehensive transportation cannot be efficiently and accurately analyzed and processed in the prior art.

[0006] In a first aspect, the present application provides a data processing method based on data consistency analysis, comprising: obtaining a plurality of structured data tables to be processed, and extracting the index names of all indexes in the plurality of structured data tables to obtain an original index name library; calculating the text similarity coefficients between the index names in the original index name library to obtain a text similarity coefficient set; determining a similar index name set of a target index name according to the text similarity coefficient set; wherein the target index name represents any index name in the original index name library; the text similarity coefficient between the index name in the similar index name set and the target index name is greater than a preset threshold; in the case where the target similar index name corresponds to the same index unit and index data magnitude as the target index name, the target similar index and the target index are merged to obtain an integrated structured data table and a benchmark index name library; wherein the target similar index name represents any index name in the similar index name set.

[0007] In an optional embodiment, extracting the index names of all indexes in the plurality of structured data tables to obtain the original index name library comprises: obtaining the level of a target index in the plurality of structured data tables; wherein the target index represents any index in all indexes; if the level of the target index is 1, the index name of the target index is stored in the original index name library; if the level of the target index is greater than 1, the target index is subjected to level merging processing, and the index name after level merging is stored in the original index name library.

[0008] In an optional embodiment, calculating the text similarity coefficients between the index names in the original index name library comprises: identifying and deleting stop words of a first index name and a second index name in the original index name library to obtain a first valid index name and a second valid index name; wherein the first index name and the second index name represent any two index names in the original index name library; subjecting the first valid index name and the second valid index name to string processing to obtain a first string and a second string; determining the number of matching characters and the number of union characters based on the first string and the second string; wherein the number of matching characters represents the number of characters with the same character and order in the first string and the second string; the number of union characters represents the number of characters after taking the union of the characters in the first string and the second string; calculating the text similarity coefficient of the first index name and the second index name based on the number of matching characters, the number of union characters, the string length of the first string, and the string length of the second string.

[0009] In an optional embodiment, the text similarity coefficient of the first index name and the second index name is calculated based on the number of matched characters, the number of union characters, the string length of the first string and the string length of the second string, including: determining the maximum value of the string length and the minimum value of the string length based on the string length of the first string and the string length of the second string; calculating the relative length of the first string and the second string based on the maximum value of the string length and the minimum value of the string length; calculating the text similarity coefficient of the first index name and the second index name based on the number of matched characters, the number of union characters and the relative length. In an optional embodiment, the target similar index and the target index are merged, including: judging whether there is compensation data in the index data of the target similar index; wherein the compensation data represents the missing index data in the target index; if there is, the compensation data is merged into the index data of the target index, and the target similar index is deleted; if not, the target similar index is deleted.

[0010] In an optional embodiment, it also includes: obtaining the unit of measurement corresponding to the target similar index name and the unit of measurement corresponding to the target index name; judging whether there is a multiple relationship between the unit of measurement corresponding to the target similar index name and the unit of measurement corresponding to the target index name; if there is, obtaining the index data of the target similar index and the index data of the target index, and performing unit uniformity on the index data to obtain first index data and second index data; wherein the first index data represents the data of the target similar index after unit uniformity; the second index data represents the data of the target index after unit uniformity; judging whether the difference between the first index data and the second index data is less than one data magnitude; if yes, it is determined that the index unit corresponding to the target similar index name and the index data magnitude are the same as the index unit corresponding to the target index name and the index data magnitude.

[0011] In an optional embodiment, the text similarity coefficient of the first index name and the second index name is calculated based on the number of matched characters, the number of union characters and the relative length, including: using the formula to calculate the text similarity coefficient of the first index name and the second index name; wherein m represents the number of matched characters, s represents the number of union characters, s1 represents the string length of the first string, s2 represents the string length of the second string, and d represents the text similarity coefficient of the first index name and the second index name.

[0012] In a second aspect, the present application provides a data processing device based on data consistency analysis, comprising: an acquisition module configured to acquire a plurality of structured data tables to be processed, and extract index names of all indexes in the plurality of structured data tables to obtain an original index name library; a calculation module configured to calculate text similarity coefficients between the index names in the original index name library to obtain a text similarity coefficient set; a determination module configured to determine a similar index name set of a target index name according to the text similarity coefficient set; wherein the target index name represents any index name in the original index name library; the text similarity coefficient between the index name in the similar index name set and the target index name is greater than a preset threshold; and a merging module configured to merge the target similar index and the target index to obtain an integrated structured data table and a benchmark index name library in a case where an index unit and an index data magnitude corresponding to a target similar index name are the same as an index unit and an index data magnitude corresponding to the target index name.

[0013] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor implements the steps of the data processing method based on data consistency analysis of any one of the preceding embodiments when executing the computer program.

[0014] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer instructions, and the computer instructions implement the data processing method based on data consistency analysis of any one of the preceding embodiments when executed by a processor.

[0015] The present application provides a data processing method based on data consistency analysis, which firstly extracts index names of all indexes from a plurality of structured data tables to construct an original index name library, then determines a similar index name set of a target index name based on text similarity coefficients between the index names, and finally merges the target index and the similar index under the premise that the index unit and the index data magnitude are consistent to obtain an integrated structured data table and a benchmark index name library. The method breaks through the data processing method based on data consistency analysis, and takes the text similarity coefficient, the index unit and the index data magnitude as references to merge data indexes with the same meaning, ensures the consistency and uniqueness of each index in the integrated structured data table, and provides support for data analysis of structured data in the comprehensive transportation field, prevents data confusion and wrong decisions, and maintains the correctness and reliability of the data system. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0017] Figure 1 A flow chart of a data processing method based on data consistency analysis provided for an embodiment of the present application is shown in FIG. 1.

[0018] Figure 2 A principle diagram for determining the number of matching characters provided for an embodiment of the present application is shown in FIG. 3.

[0019] Figure 3 A functional module diagram of a data processing device based on data consistency analysis provided for an embodiment of the present application is shown in FIG. 4.

[0020] Figure 4 A schematic diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art without creative effort on the basis of the embodiments in the present application belong to the scope of protection of the present application.

[0023] The following will make a detailed description of some embodiments of the present application in combination with the drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] The consistency of the basic structured data in the field of comprehensive transportation mainly includes the uniformity of naming, that is, ensuring the same type of data uses the same naming rules at different data sources and different time points; and the uniformity of data structure, ensuring the data model and architecture remain consistent at different data sources and different time points. At present, there is no efficient, convenient and accurate automatic method for analyzing the consistency of structured data in this field. Manual analysis has problems such as poor accuracy and low efficiency, and cannot meet the actual demand of a large amount of data aggregation and updating. Therefore, the embodiment of the present application provides a data processing method based on data consistency analysis to alleviate the technical problems involved above.

[0025] Embodiment one

[0026] Figure 1 A flowchart of the data processing method based on data consistency analysis provided by the embodiment of the present application is shown in Figure 1 The method specifically includes the following steps:

[0027] In step S102, a plurality of structured data tables to be processed are obtained, and the index names of all indexes in the plurality of structured data tables are extracted to obtain an original index name library.

[0028] The method provided by the embodiment of the present application can be used to analyze and process the basic structured data in various statistical yearbooks in the field of comprehensive transportation, such as the basic structured data in the mainstream statistical yearbooks in the field of comprehensive transportation, such as Highway Maintenance Annual Report, China Highway Network Operation Blue Book and International Statistical Yearbook. Generally, the data processing object of the embodiment of the present application is a plurality of structured data tables obtained from one data interface, which can be understood as the same data source.

[0029] Due to differences in different data statisticians / data statistics time, data indexes (referred to as indexes) with the same physical meaning may use different index names. Therefore, after obtaining the plurality of structured data tables, the index names of all indexes are first extracted from the plurality of structured data tables to construct an original index name library, and then the plurality of structured data tables are integrated through consistency analysis of the index names in the original index name library.

[0030] In step S104, the text similarity coefficients between the index names in the original index name library are calculated to obtain a text similarity coefficient set.

[0031] The text similarity coefficient between the index names can represent the similarity and difference between the index names, and therefore, the measurement of the consistency of the index names is determined by the text similarity coefficient between the index names. If the text similarity coefficient between the index name A and the index name B is very high, it is considered that the consistency of the two is better. Therefore, after the original index name library is determined, the text similarity coefficient between any two index names in the original index name library is further calculated, and the text similarity coefficient set is obtained after the traversal is completed, that is, the set composed of the calculation results of the text similarity coefficients between all the index names.

[0032] In step S106, the similar index name set of the target index name is determined according to the text similarity coefficient set.

[0033] The target index name represents any index name in the original index name library; the text similarity coefficient between the index name in the similar index name set and the target index name is greater than the preset threshold.

[0034] After the text similarity coefficient set is obtained by calculation, if the text similarity coefficient between a certain index name in the original index name library and the target index name is greater than the preset threshold, the index name can be called the similar index name of the target index name. Therefore, by traversing the calculation results of the text similarity coefficients between the target index name and other index names, the similar index name set of the target index name can be determined by screening with the preset threshold. The value of the preset threshold is not specifically limited in the embodiment of the application, and the user can select according to the actual situation. Optionally, the preset threshold is 50%.

[0035] In step S108, the target similar index and the target index are merged to obtain the integrated structured data table and the benchmark index name library in the case that the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name.

[0036] The target similar index name represents any index name in the similar index name set.

[0037] Specifically, in order to ensure the accuracy of the index merging, the embodiment of the application does not directly merge the index data of different indexes with the text similarity coefficient greater than the preset threshold, but further needs to consider the index unit and the index data magnitude of the target index and the target similar index. If the index unit and / or the index data magnitude of the target index and the target similar index are different, it is determined that the target index and the target similar index are different indexes, that is, the two cannot be merged.

[0038] If the index unit and the index data magnitude of the target index and the target similar index are the same, it is determined that the target index and the target similar index are the same index, in which case, the target similar index and the target index need to be merged to ensure the uniqueness of the physical meaning of each index, the merging of the index includes the merging of the index name and the merging of the index data, the merging of the index name can be randomly selecting one of the two merged index names to keep, or can be selecting the index name with more or less characters according to the demand; the merging of the index data is taking the union set of the index data. After all the similar indexes are merged, the integrated structured data table and the benchmark index name library are obtained. As known from the above introduction, the benchmark index name library is the result of deleting part of the index names from the original index name library, and each index name in the benchmark index name library represents an index with uniqueness.

[0039] The embodiment of the present application provides a data processing method based on data consistency analysis, which firstly extracts the index names of all indexes from a plurality of structured data tables to construct an original index name library, then determines the similar index name set of the target index name based on the text similarity coefficient between the index names, and finally merges the target index and the similar index under the premise that the index unit and the index data magnitude are consistent, to obtain the integrated structured data table and the benchmark index name library. The method breaks through the data processing method based on data consistency analysis, and takes the text similarity coefficient, the index unit and the index data magnitude as the reference to merge the data indexes with the same meaning, to ensure the consistency and uniqueness of each index in the integrated structured data table, to provide support for the data analysis of the structured data in the comprehensive transportation field, to prevent data confusion and wrong decision, and to maintain the correctness and reliability of the data system.

[0040] In an optional embodiment, in the step S102, the index names of all indexes in the plurality of structured data tables are extracted to obtain the original index name library, which specifically includes the following contents:

[0041] The level of the target index in the plurality of structured data tables is obtained; wherein the target index represents any index in all indexes.

[0042] If the level of the target index is 1, the index name of the target index is stored in the original index name library; if the level of the target index is greater than 1, the target index is subjected to level merging processing, and the index name after the level merging is stored in the original index name library.

[0043] For the convenience of understanding, the extraction of the index name is exemplified below. Table 1 is an index name part of a structured data table. In Table 1, both indexes with a level of 2 and indexes with a level of 1 are included. The index "Road section, bridge line camera" has a level of 1, and thus the index name "Road section, bridge line camera" of the index is stored in the original index name library. The index "Cumulative since the beginning of the year" has a level of 2, and thus the index needs to be combined in levels. If the connection symbol between the levels is connected by ":", the index name after the level combination is "Highway passenger volume: cumulative since the beginning of the year", that is, arranged in the order from the top level to the bottom level. The embodiment of the present application does not specifically limit the connection symbol between the levels when the levels are combined, and any symbol can also be selected not to be connected.

[0044] It should be noted that if each index of the structured data table is also marked with a corresponding unit of measurement, for example, year-on-year growth rate (%), road section, bridge line camera (number), then the starting character of the parentheses "(" can be set as a stop extraction identifier, because the parentheses and the characters inside the parentheses are not index names.

[0045] Table 1 index name of structured data table

[0046]

[0047] In an optional embodiment, the step S104 of calculating the text similarity coefficient between the index names in the original index name library specifically includes the following steps:

[0048] In step S1041, stop words in the first index name and the second index name in the original index name library are identified and deleted to obtain a first valid index name and a second valid index name. The first index name and the second index name represent any two index names in the original index name library.

[0049] In step S1042, the first valid index name and the second valid index name are subjected to string processing to obtain a first string and a second string.

[0050] In step S1043, based on the first string and the second string, the number of matching characters and the number of union characters are determined. The number of matching characters represents the number of characters with the same character and order in the first string and the second string. The number of union characters represents the number of characters after the characters in the first string and the second string are taken in union.

[0051] In step S1044, based on the number of matching characters, the number of union characters, the string length of the first string and the string length of the second string, the text similarity coefficient of the first index name and the second index name is calculated.

[0052] Specifically, to calculate the text similarity between the index names, the embodiment of the present application needs to perform string processing (only keep the text) on all index names in the original index name library, and before the string processing, it is necessary to first perform stop word filtering to remove words that have no actual meaning or are insignificant in natural language processing, such as “of”, “is”, “has”, etc., so as to improve the accuracy and efficiency of the subsequent text similarity coefficient calculation. The embodiment of the present application does not specifically limit the order of stop word filtering and string processing, and string processing can be performed first, and then stop word filtering.

[0053] The following Table 2 exemplarily shows the results of string processing on part of the index names.

[0054] Table 2: Index names and corresponding strings

[0055]

[0056] To calculate the text similarity coefficient between the first index name and the second index name in the original index name library, the number of matching characters and the number of union characters of the two should be determined after the first index name and the second index name are processed by removing stop words and string processing. Figure 2 A principle diagram for determining the number of matching characters provided by the embodiment of the present application. Taking the first string as {road, section, bridge, beam, along, line, camera, image, machine} and the second string as {road, section, bridge, beam, along, line, camera, image, machine, number, quantity} as an example, according to the characters and the order of the characters in the string, the characters that are the same in the first string and the second string are {road, section, bridge, beam, along, line, camera, image, machine}, that is, the number of matching characters is 9. And the union of the characters of the first string and the second string is {road, section, bridge, beam, along, line, camera, image, machine, number, quantity}, that is, the number of union characters is 11.

[0057] After the number of matching characters and the number of union characters of the first string and the second string are determined, the text similarity coefficient of the first index name and the second index name can be calculated in combination with the string length of the first string and the string length of the second string.

[0058] In an optional embodiment, the step S1044 of calculating the text similarity coefficient of the first index name and the second index name based on the number of matching characters, the number of union characters, the string length of the first string and the string length of the second string specifically includes the following steps:

[0059] Step S10441, determine the maximum value of the string length and the minimum value of the string length based on the string length of the first string and the string length of the second string.

[0060] In step S10442, the relative length of the first string and the second string is calculated based on the maximum value of the string length and the minimum value of the string length.

[0061] In step S10443, the text similarity coefficient of the first index name and the second index name is calculated based on the number of matching characters, the number of union characters and the relative length.

[0062] In the embodiment of the present application, the calculation method of the text similarity coefficient between index names is based on the ratio of the number of matching characters to the number of union characters, and the relative length of the two strings is also considered. Alternatively, the text similarity coefficient of the first index name and the second index name is calculated based on the number of matching characters, the number of union characters and the relative length, including:

[0063] The text similarity coefficient of the first index name and the second index name is calculated by using the formula ; wherein m represents the number of matching characters, s represents the number of union characters, s1 represents the string length of the first string, s2 represents the string length of the second string, and d represents the text similarity coefficient of the first index name and the second index name.

[0064] For example, the first index name "Road section, bridge along the line camera" corresponds to the first string r1=(road, section, bridge, along, line, camera, machine), and the string length of the first string s1=9; the second index name "Road section, bridge along the line camera: quantity" corresponds to the second string r2=(road, section, bridge, along, line, camera, machine, quantity), and the string length of the second string s2=11. By comparing r1 and r2, the number of matching characters m=9. The number of union characters s=11. Comparing the string lengths, max(s1, s2) is 11, and min(s1, s2) is 9. Therefore, the text similarity coefficient of the first index name and the second index name d=66.94%. Table 3 below is a table showing the calculation results of the text similarity coefficient between index names.

[0065] Table 3 Calculation results of the text similarity coefficient between index names

[0066]

[0067]

[0068] If the threshold of the text similarity coefficient of the infrastructure data index name in the comprehensive transportation field is set as 50%, that is, greater than or equal to 50% represents that the two index names have high similarity. For example, in Table 3, "Road section, bridge along the camera" and "Road section, bridge along the camera: quantity" have high similarity, and "Postal industry total volume: express business: international" and "Postal industry delivery volume: express business: international" have high similarity.

[0069] In an optional embodiment, the merging of the target similar index and the target index in step S108 specifically includes the following steps.

[0070] Step S1081, determining whether there is compensation data in the index data of the target similar index; wherein the compensation data represents the missing index data in the target index.

[0071] If there is, the following step S1082 is performed; if not, the following step S1083 is performed.

[0072] Step S1082, merging the compensation data into the index data of the target index, and deleting the target similar index.

[0073] Step S1083, deleting the target similar index.

[0074] After determining that the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name, the target similar index and the target index can be merged. Specifically, taking the index data of the target index as a reference, it is determined whether there is index data (i.e., compensation data) in the index data of the target similar index that is not in the target index. If there is, the compensation data is merged into the index data of the target index. After the index data is merged, the target similar index does not need to be retained. If there is no compensation data in the index data of the target similar index, it means that the index data of the target similar index is already included in the index data of the target index. Therefore, in order to ensure the uniqueness of the index, the target similar index is deleted.

[0075] For example, the statistical years corresponding to the index data of the target index are 2000-2003, and the statistical years corresponding to the index data of the target similar index are 2004-2006. Obviously, there is compensation data in the index data of the target similar index. The compensation data is merged into the index data of the target index, and the updated index data of the target index corresponds to the statistical years 2000-2006.

[0076] In an alternative embodiment, in order to determine whether the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name, the embodiments of the present application further include the following contents:

[0077] The measurement unit corresponding to the target similar index name and the measurement unit corresponding to the target index name are obtained.

[0078] It is determined whether there is a multiple relationship between the measurement unit corresponding to the target similar index name and the measurement unit corresponding to the target index name.

[0079] If there is, the index data of the target similar index and the index data of the target index are obtained, and the unit of the index data is unified to obtain first index data and second index data; wherein the first index data represents the index data of the target similar index after unit unification; and the second index data represents the index data of the target index after unit unification.

[0080] It is determined whether the difference between the first index data and the second index data is less than one data magnitude.

[0081] If yes, it is determined that the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name.

[0082] Specifically, taking the target index name as "road section, bridge along the camera", and the target similar index name as "road section, bridge along the camera: quantity" as an example, by querying the structured data table, it can be determined that the measurement unit of "road section, bridge along the camera" is "set", and the measurement unit of "road section, bridge along the camera: quantity" is "piece", and since there is no multiple relationship between the measurement units of the two, the index data of the above two indexes is abandoned.

[0083] Taking the target index name as "total volume of postal industry business: express business: international", and the target similar index name as "volume of postal industry delivery business: express business: international" as an example, by querying the structured data table, it is determined that the measurement units of the two are both "ten thousand pieces", that is, there is a multiple relationship (the multiple is 1) between the measurement units, and then the determination of whether the difference between the index data is less than one data magnitude is continued.

[0084] Data magnitude refers to the scale or size level of quantity, and a fixed ratio is maintained between adjacent data magnitudes. The commonly used ratios are 10, 2, 1000, 1024, and e (the base of natural logarithm). According to the research on the data samples in the field of comprehensive transportation, the optional value of the ratio between adjacent data magnitudes of the index data in the field of comprehensive transportation is 10. That is, if the first index data and the second index data are divided, the result is between 0.1 (the inverse of the ratio between data magnitudes) and 10 (the ratio between data magnitudes), that is, it is determined that the difference between the two is less than one data magnitude. For example, 99 divided by 100 is 0.99, which is within the interval of 0.1 to 10, that is, the difference between the two is less than one data magnitude; 1100 divided by 100 is 11, which is outside the interval of 0.1 to 10, that is, it is determined that the difference between the two is greater than one data magnitude.

[0085] If the multiple is 1, the unit is unified before and after, the index data is unchanged, and it is determined whether the difference between the first index data and the second index data is less than one data magnitude. If the difference is less than one data magnitude, it is determined that the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name. In the embodiment of the present application, the index unit refers to the measurement unit without data magnitude, for example, the measurement unit "ten thousand pieces", in which the data magnitude is "ten thousand", and after removing the data magnitude, the corresponding index unit is "piece".

[0086] Continuing to take the target index name "total volume of postal industry: express business: international" and the target similar index name "volume of postal industry: express business: international" as an example, if the measurement unit of "total volume of postal industry: express business: international" is "piece" and the measurement unit of "volume of postal industry: express business: international" is "ten thousand pieces" by querying the structured data table, it can be known that there is a multiple relationship between the measurement units, and the multiple is ten thousand.

[0087] In the case of a multiple greater than 1, the index data of the target similar index and the index data of the target index need to be unified in unit before judging whether the difference between them is less than a data magnitude, for example, the index data of "total volume of postal industry: express business: international" is 10000, and the measurement unit is piece, while the index data of "volume of postal industry delivery business: express business: international" is 2, and the measurement unit is ten thousand pieces, if the units are unified to pieces, then the index data after unification is 10000 and 20000 respectively, by comparing the ratio between the ratio 0.5 and the pre-set data magnitude (i.e. 10), since 0.5 is within the interval (0.1, 10), it is determined that the difference between them is less than a data magnitude, and it is determined that the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name.

[0088] In summary, the embodiment of the application provides a complete data processing method based on data consistency analysis, which can calculate the similarity of index names between various data in the field of comprehensive transportation, and make data specification judgment on the basis of the similarity, merge data indexes with the same meaning and the same statistical approach, ensure the consistency and uniqueness of each index, provide support for subsequent data analysis and use of the transportation industry data, prevent data confusion and wrong decision, and maintain the correctness and reliability of the data system.

[0089] Embodiment two

[0090] The embodiment of the application also provides a data processing device based on data consistency analysis, which is mainly used for executing the data processing method based on data consistency analysis provided in the above embodiment one, and the data processing device based on data consistency analysis provided by the embodiment of the application will be specifically introduced below.

[0091] Figure 3 The function module diagram of the data processing device based on data consistency analysis provided by the embodiment of the application is shown in the figure Figure 3 As shown in the figure, the device mainly comprises: an acquisition module 10, a calculation module 20, a determination module 30 and a merging module 40, wherein: the acquisition module 10 is used for acquiring a plurality of structured data tables to be processed, and extracting the index names of all indexes in the plurality of structured data tables to obtain an original index name library.

[0092] The calculation module 20 is used for calculating the text similarity coefficients between the index names in the original index name library to obtain a text similarity coefficient set.

[0093] The determining module 30 is configured to determine a similar index name set of the target index name according to the text similarity coefficient set; wherein the target index name represents any index name in the original index name library; and the text similarity coefficient between the index name in the similar index name set and the target index name is greater than a preset threshold.

[0094] The merging module 40 is configured to merge the target similar index and the target index to obtain the integrated structured data table and the benchmark index name library in the case that the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name; wherein the target similar index name represents any index name in the similar index name set.

[0095] The embodiment of the present application provides a data processing device based on data consistency analysis. The device firstly extracts index names of all indexes from a plurality of structured data tables to construct an original index name library, then determines a similar index name set for a target index name based on a text similarity coefficient between index names, and finally merges the target index and the similar index under the premise that the index unit and the index data magnitude are consistent to obtain an integrated structured data table and a benchmark index name library. The device breaks through a data processing flow based on data consistency analysis, takes the text similarity coefficient, the index unit and the index data magnitude as references, merges data indexes with the same meaning, guarantees the consistency and uniqueness of each index in the integrated structured data table, provides support for data analysis of structured data in the comprehensive transportation field, prevents data confusion and wrong decisions, and maintains the correctness and reliability of the data system.

[0096] Optionally, the obtaining module 10 is specifically configured to:

[0097] Obtain the level of the target index in the plurality of structured data tables; wherein the target index represents any index in all indexes.

[0098] If the level of the target index is 1, the index name of the target index is stored in the original index name library.

[0099] If the level of the target index is greater than 1, the target index is subjected to level merging processing, and the index name after the level merging is stored in the original index name library.

[0100] Optionally, the calculating module 20 comprises:

[0101] The identifying and deleting unit is configured to identify and delete stop words of the first index name and the second index name in the original index name library to obtain the first valid index name and the second valid index name; wherein the first index name and the second index name represent any two index names in the original index name library.

[0102] The processing unit is configured to perform string processing on the first valid index name and the second valid index name to obtain a first string and a second string.

[0103] The determining unit is configured to determine a matching character quantity and a union character quantity based on the first string and the second string, wherein the matching character quantity represents a number of characters that are identical in character and order in the first string and the second string, and the union character quantity represents a number of characters after taking a union of characters in the first string and the second string.

[0104] The calculating unit is configured to calculate a text similarity coefficient of the first index name and the second index name based on the matching character quantity, the union character quantity, a string length of the first string, and a string length of the second string.

[0105] Optionally, the calculating unit includes:

[0106] The determining subunit is configured to determine a maximum value of the string lengths and a minimum value of the string lengths based on the string length of the first string and the string length of the second string.

[0107] The first calculating subunit is configured to calculate a relative length of the first string and the second string based on the maximum value of the string lengths and the minimum value of the string lengths.

[0108] The second calculating subunit is configured to calculate the text similarity coefficient of the first index name and the second index name based on the matching character quantity, the union character quantity, and the relative length.

[0109] Optionally, the merging module 40 is specifically configured to:

[0110] determine whether compensation data exists in index data of the target similar index, wherein the compensation data represents missing index data in the target index.

[0111] If the compensation data exists, the compensation data is merged into the index data of the target index, and the target similar index is deleted.

[0112] If the compensation data does not exist, the target similar index is deleted.

[0113] Optionally, the apparatus is further configured to:

[0114] obtain a unit of measurement corresponding to the target similar index name and a unit of measurement corresponding to the target index name.

[0115] determine whether a multiple relationship exists between the unit of measurement corresponding to the target similar index name and the unit of measurement corresponding to the target index name.

[0116] If the target similar index exists, the index data of the target similar index and the index data of the target index are acquired, and the index data is unified in units to obtain first index data and second index data; wherein the first index data represents the index data of the target similar index after unit unification; and the second index data represents the index data of the target index after unit unification.

[0117] It is judged whether the difference between the first index data and the second index data is less than a data magnitude.

[0118] If yes, it is determined that the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name.

[0119] Optionally, the second calculation subunit is specifically used for:

[0120] The text similarity coefficient of the first index name and the second index name is calculated by using an algorithm. The relative length is represented by s1 / s2, wherein s1 represents the string length of the first string, s2 represents the string length of the second string, and d represents the text similarity coefficient of the first index name and the second index name.

[0121] Embodiment three

[0122] Referring to Figure 4 , the embodiment of the present application provides an electronic device, which comprises a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is used for executing an executable module stored in the memory 61, such as a computer program.

[0123] The memory 61 can include a high-speed random access memory (RAM, Random Access Memory) and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0124] The bus 62 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 4 only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus. ​

[0125] The memory 61 is configured to store a program, and the processor 60 is configured to execute the program after receiving an execution instruction. The method executed by the device defined by the process disclosed in any of the embodiments of the present application can be applied to the processor 60, or implemented by the processor 60.

[0126] The processor 60 can be an integrated circuit chip having a processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 60 or the instruction in the form of software. The processor 60 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61, and combines the hardware to complete the steps of the above method.

[0127] The computer program product of the data processing method, device and electronic equipment based on data consistency analysis provided by the embodiments of the present application includes a computer readable storage medium storing non-volatile program codes executable by a processor. The instructions included in the program codes can be used to execute the method in the foregoing method embodiments. For specific implementation, please refer to the method embodiments, which will not be described here.

[0128] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0129] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0130] It should be noted that similar reference numerals and letters refer to similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0131] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0132] In addition, the terms "first", "second", "third", etc. are only used for differentiation and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", "overhanging", etc. do not mean that the components must be absolutely horizontal or overhanging, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0133] In the description of the present application, it should also be noted that unless otherwise explicitly specified and limited, the terms "arrangement", "installation", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0134] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method based on data consistency analysis, characterized by, The method comprises the following steps: obtaining a plurality of structured data tables to be processed, and extracting index names of all indexes in the plurality of structured data tables to obtain an original index name library; calculating text similarity coefficients between index names in the original index name library to obtain a text similarity coefficient set; determining a similar index name set of a target index name according to the text similarity coefficient set; wherein the target index name represents any index name in the original index name library; the text similarity coefficient between the index name in the similar index name set and the target index name is greater than a preset threshold value; in a case where the index unit corresponding to a target similar index name and the index data magnitude corresponding to the target similar index name are the same as the index unit corresponding to the target index name and the index data magnitude corresponding to the target index name, respectively, the target similar index and the target index are merged to obtain an integrated structured data table and a benchmark index name library; wherein the target similar index name represents any index name in the similar index name set; extracting index names of all indexes in the plurality of structured data tables to obtain an original index name library, comprising: obtaining a level of a target index in the plurality of structured data tables; wherein the target index represents any index in the all indexes; if the level of the target index is 1, the index name of the target index is stored in the original index name library; if the level of the target index is greater than 1, the target index is subjected to level merging processing, and the index name after level merging is stored in the original index name library; calculating text similarity coefficients between index names in the original index name library, comprising: identifying and deleting stop words of a first index name and a second index name in the original index name library to obtain a first valid index name and a second valid index name; wherein the first index name and the second index name represent any two index names in the original index name library; subjecting the first valid index name and the second valid index name to string processing to obtain a first string and a second string; determining a matching character number and a union character number based on the first string and the second string; wherein the matching character number represents the number of characters that are the same in character and order in the first string and the second string; the union character number represents the number of characters after taking the union of the characters in the first string and the second string; calculating a text similarity coefficient of the first index name and the second index name based on the matching character number, the union character number, the string length of the first string and the string length of the second string; calculating a text similarity coefficient of the first index name and the second index name based on the matching character number, the union character number, the string length of the first string and the string length of the second string, comprising: determining a maximum value of string length and a minimum value of string length based on the string length of the first string and the string length of the second string; calculating a relative length of the first string and the second string based on the maximum value of the string length and the minimum value of the string length; calculating a text similarity coefficient of the first index name and the second index name based on the number of matched characters, the number of characters in the union set and the relative length.

2. The data processing method based on data consistency analysis according to claim 1, characterized in that, merging the target similar index and the target index, including: judging whether there is compensation data in the index data of the target similar index; wherein the compensation data represents the missing index data in the target index; if there is, merging the compensation data into the index data of the target index and deleting the target similar index; if there is not, deleting the target similar index.

3. The data processing method based on data consistency analysis according to claim 1, characterized in that, further comprising: obtaining the unit of measurement corresponding to the target similar index name and the unit of measurement corresponding to the target index name; judging whether there is a multiple relationship between the unit of measurement corresponding to the target similar index name and the unit of measurement corresponding to the target index name; if there is, obtaining the index data of the target similar index and the index data of the target index, and performing unit unification on the index data to obtain first index data and second index data; wherein the first index data represents the index data of the target similar index after unit unification; the second index data represents the index data of the target index after unit unification; judging whether the difference between the first index data and the second index data is less than one data magnitude; if yes, determining that the index unit and the index data magnitude corresponding to the target similar index name are the same as the index unit and the index data magnitude corresponding to the target index name.

4. The data processing method based on data consistency analysis according to claim 1, characterized in that, calculating a text similarity coefficient of the first index name and the second index name based on the number of matched characters, the number of characters in the union set and the relative length, including: Utilizing the equation calculating the text similarity coefficient of the first index name and the second index name; wherein m represents the number of matching characters, s represents the number of union characters, represents the relative length, s1 represents the string length of the first string, s2 represents the string length of the second string, and d represents the text similarity coefficient of the first index name and the second index name.

5. The data processing method based on data consistency analysis according to claim 1, characterized in that, if each index of the structured data table is also marked with the corresponding unit of measurement, the starting character of the bracket is set as the stop extraction identifier.

6. The data processing method based on data consistency analysis according to claim 1, characterized in that, if the text similarity coefficient is greater than or equal to 50%, it represents that the two index names have high similarity.

Images