Municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence

Abstract

The invention discloses a municipal road project cost data cleaning and preprocessing method based on artificial intelligence, and the method comprises the following steps: 1, setting project index characteristic parameters, and the specific type of each project index characteristic parameter; 2, key information is acquired from the project general situation text for preliminary arrangement; 3, performing clustering analysis, and identifying and removing repeated data records; 4, analyzing the missing data to deduce the missing data; 5, performing matching verification on missing data obtained by deduction; 6, performing standardization or normalization processing on the data by using a neural network; 7, integrating all missing data obtained by deduction and all data after standardization or normalization processing to form a new feature combination in a unified standard format; and screening out the most representative feature combination by using a weight evaluation method of feature parameters. According to the method, while the data quality is ensured, the prediction capability and the analysis value of the data are enhanced, so that the process is more efficient and reliable.

Description

Technical Field

[0001] The present invention relates to the technical field of road engineering cost data cleaning and preprocessing, and in particular to a municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence. Background Art

[0002] In the field of engineering cost, data cleaning and preprocessing are the crucial first steps in data analysis and machine learning. Municipal road projects usually involve a large amount of data, including design drawings, construction records, material lists, cost accounting, etc. These data are in various formats and from a wide range of sources, and contain a large amount of text, digital and image information, which increases the difficulty of data cleaning and preprocessing.

[0003] Professional terminology places higher demands on data cleaning and preprocessing. Municipal road engineering data is updated frequently and may contain missing values ​​and outliers.

[0004] Existing technologies often rely on manual data cleaning and preprocessing, such as manual identification and extraction of key information, manual filling of missing values, etc., which are difficult to adapt to the processing needs of large amounts of data. Manual participation can also easily affect data quality and analysis accuracy.

[0005] Therefore, how to identify and remove duplicates while ensuring the consistency or similarity of the data, and perform semantic analysis on this basis, adjust the scale of the data, accumulate data as a verification data set, and verify the effectiveness of data cleaning and preprocessing has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the present invention provides a municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence, the purpose of which is to automatically generate new feature combinations by binding feature parameters and parameter values ​​with source data, and integrate them into unified standard parameter values. While ensuring data quality, it enhances the predictive ability and analytical value of the data, improves the efficiency of data processing, reduces manual intervention, and makes the process of data analysis and machine learning more efficient and reliable.

[0007] To achieve the above object, the present invention discloses a municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence, comprising the following steps:

[0008] Step 1: setting engineering indicator characteristic parameters for municipal road engineering cost data that needs data cleaning and pre-processing, and the specific type of each engineering indicator characteristic parameter;

[0009] The engineering indicator characteristic parameters include road length, road width, number of lanes, section name and pavement material;

[0010] Step 2: Obtaining original data from the project overview text for keyword matching to identify text paragraphs related to the project indicator characteristic parameters;

[0011] and extracting key information from the text paragraph;

[0012] Then, the key information is preliminarily sorted out;

[0013] Step 3: similar data deduplication and outlier detection; specifically: using a density clustering algorithm to perform cluster analysis on the key information that has been preliminarily sorted, identifying and removing duplicate data records;

[0014] Step 4: Analyze the missing data for all missing characteristic parameter value parts of the engineering indicator characteristic parameters, specifically: infer the missing data by identifying context information or the non-missing characteristic parameter values;

[0015] Step 5: performing matching verification on the missing data obtained by inference;

[0016] If the matching verification is unsuccessful, re-execute steps 2 to 4;

[0017] If the matching verification is successful, continue to execute the subsequent steps;

[0018] Step 6: According to the engineering indicator characteristic parameters, the characteristic parameters that can be successfully matched in the engineering overview text are standardized or normalized using a neural network;

[0019] Step 7: Integrate all the missing data obtained by inference and all the data after the standardization or normalization process to form a new feature combination in a unified standard format;

[0020] Then, the weight evaluation method of feature parameters is used to screen out the most representative feature combination.

[0021] Preferably, the specific types of the road length, the road width and the number of lanes are all numerical types;

[0022] The specific types described in the bid section names are all text types;

[0023] The specific types of pavement materials described are all enumerated.

[0024] Preferably, the pavement material includes concrete and asphalt.

[0025] Preferably, in step 2, natural language processing (NLP) technology is used to perform keyword matching on the original data obtained from the engineering overview text.

[0026] Preferably, in step 2, the key information is initially sorted, specifically including removing irrelevant characters and converting into a unified format.

[0027] Preferably, in step 3, based on the clustering results and statistical methods, data points that deviate from the normal range in the key information that has been preliminarily sorted are identified and marked as outliers, and then all the outliers are deleted or corrected based on other data points;

[0028] Alternatively, similarity calculation is used to identify and remove duplicates, and unmatched data is considered as outliers and discarded without further identification and matching.

[0029] Preferably, in step 4, a deep learning model is constructed to predict the missing data, or an existing data training model is used to predict;

[0030] And according to the prediction results, a suitable strategy is selected to fill the missing data by directly using the predicted value or taking the average value or median method according to the data type.

[0031] Preferably, in step 6, a Min-Max Scaling or Z-Score normalized neural network is used to standardize or normalize the data.

[0032] Preferably, step 7 automatically generates a new feature combination in a unified standard format based on the engineering indicator characteristic parameters, inferring all the missing data obtained, and all the data after the standardization or normalization processing.

[0033] Beneficial effects of the present invention:

[0034] While ensuring data quality, the present invention enhances the predictive ability and analytical value of data, improves the efficiency of data processing, reduces human intervention, and makes the process of data analysis and machine learning more efficient and reliable.

[0035] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flow chart showing an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0037] Example

[0038] like Figure 1 As shown, the municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence includes the following steps:

[0039] Step 1: Set engineering indicator characteristic parameters for municipal road engineering cost data that needs data cleaning and pre-processing, as well as the specific type of each engineering indicator characteristic parameter;

[0040] The characteristic parameters of engineering indicators include road length, road width, number of lanes, section name and pavement material;

[0041] Step 1 is used to establish characteristic parameters and parameter value type standards, clarify the specific types of parameter value types, and different parameter value types determine the accuracy of the characteristic parameter values ​​that can be matched subsequently.

[0042] Step 1 for the present invention is to ensure complete coverage of characteristic parameters and corresponding parameter value types, and to ensure consistency or similarity between engineering files, engineering overview texts and characteristic integral parameters and parameter values.

[0043] Step 2: Obtaining original data from the project overview text for keyword matching to identify text paragraphs related to the project indicator characteristic parameters;

[0044] and extract key information from text paragraphs;

[0045] Then, conduct preliminary sorting of key information;

[0046] Step 2 for the present invention is to accumulate a portion of data as a verification data set to verify the effects of data cleaning and preprocessing.

[0047] Step 3: Deduplication of similar data and detection of outliers; specifically: use density clustering algorithm to perform cluster analysis on the key information that has been initially sorted, identify and remove duplicate data records;

[0048] Step 4: Analyze the missing data for all missing characteristic parameter values ​​of the engineering indicator characteristic parameters, specifically: infer the missing data by identifying context information or non-missing characteristic parameter values;

[0049] Step 5: Match and verify the missing data obtained by inference;

[0050] If the matching verification fails, repeat steps 2 to 4;

[0051] If the match verification is successful, continue to the subsequent steps;

[0052] Step 6: According to the engineering indicator characteristic parameters, the characteristic parameters that can be successfully matched in the engineering overview text are standardized or normalized using a neural network;

[0053] Step 7: Integrate all missing data obtained by inference and all data after standardization or normalization to form a new feature combination in a unified standard format;

[0054] Then, the weight evaluation method of feature parameters is used to screen out the most representative feature combination.

[0055] The present invention solves the problems existing in the prior art by extracting characteristic parameters and parameter values ​​from actual engineering documents and engineering overview texts, matching them according to corresponding types of system standard characteristic parameters and parameter values, and automatically filling in and supplementing missing characteristic parameter values, thereby improving the efficiency and accuracy of data cleaning and preprocessing.

[0056] The present invention combines semantically similar feature parameters together, and when similar feature parameters and parameter values ​​are recognized again, the feature parameter values ​​of the corresponding type will be automatically filled in. Of course, the scale of adjusting the data can be adjusted according to actual needs.

[0057] In some embodiments, the specific types of the road length, road width, and number of lanes are all numerical types;

[0058] The specific types of the bid section names are all text type;

[0059] The specific types of pavement materials are all enumerated.

[0060] In certain embodiments, the pavement material includes concrete and asphalt.

[0061] In some embodiments, in step 2, natural language processing (NLP) technology is used to perform keyword matching on the raw data obtained from the engineering overview text.

[0062] In some embodiments, in step 2, the key information is initially sorted out, specifically including removing irrelevant characters and converting into a unified format.

[0063] In some embodiments, in step 3, based on the clustering results and statistical methods, data points that deviate from the normal range in the key information that has been initially sorted are identified and marked as outliers, and then all outliers are deleted or corrected based on other data points;

[0064] Alternatively, similarity calculation is used to identify and remove duplicates, and unmatched data is considered as outliers and discarded without further identification and matching.

[0065] In some embodiments, in step 4, a deep learning model is constructed to predict missing data, or a model is trained using existing data to predict;

[0066] And according to the prediction results, choose the appropriate strategy to fill the missing data by directly using the predicted value or taking the average or median method according to the data type.

[0067] For example, the text of the project overview: "Road accurate to branch road" has been matched to "road" and "branch road". The keyword "branch road" can be associated with the road grade, and the match is successful.

[0068] In some embodiments, in step 6, the data is standardized or normalized using a Min-Max Scaling or Z-Score normalization neural network.

[0069] In practical applications, neural networks can learn the distribution characteristics of data and form memory points so that they can quickly process similar data in the future. Example: Project Overview Text: "Road Accurate to Branch Road", "Road Accurate to" is a set of characteristic parameters in the current project overview text, "branch road" is the characteristic value, and similar arrangements and text descriptions.

[0070] In some embodiments, step 7 automatically generates a new feature combination in a unified standard format based on the engineering indicator feature parameters, inferring all missing data, and all data after standardization or normalization.

[0071] In practical applications, the above technical means can enhance the predictive ability and analytical value of data.

[0072] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence; characterized in that: The steps include: Step 1: setting engineering indicator characteristic parameters for municipal road engineering cost data that needs data cleaning and pre-processing, as well as the specific type of each engineering indicator characteristic parameter; The engineering indicator characteristic parameters include road length, road width, number of lanes, section name and pavement material; Step 2: Obtaining original data from the project overview text for keyword matching to identify text paragraphs related to the project indicator characteristic parameters; and extracting key information from the text paragraph; Then, the key information is preliminarily sorted out; Step 3: similar data deduplication and outlier detection; specifically: using a density clustering algorithm to perform cluster analysis on the key information that has been preliminarily sorted, identifying and removing duplicate data records; Step 4: Analyze the missing data for all missing characteristic parameter value parts of the engineering indicator characteristic parameters, specifically: infer the missing data by identifying context information or the non-missing characteristic parameter values; Step 5: performing matching verification on the missing data obtained by inference; If the matching verification is unsuccessful, re-execute steps 2 to 4; If the matching verification is successful, continue to execute the subsequent steps; Step 6: According to the engineering indicator characteristic parameters, the characteristic parameters that can be successfully matched in the engineering overview text are standardized or normalized using a neural network; Step 7: Integrate all the missing data obtained by inference and all the data after the standardization or normalization process to form a new feature combination in a unified standard format; Then, the weight evaluation method of feature parameters is used to screen out the most representative feature combination.

2. The method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence according to claim 1 is characterized in that: The specific types of the road length, the road width and the number of lanes are all numerical types; The specific types described in the bid section names are all text types; The specific types of pavement materials described are all enumerated.

3. The method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence according to claim 1 is characterized in that: The pavement materials include concrete and asphalt.

4. The method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence according to claim 1 is characterized in that: In step 2, natural language processing (NLP) technology is used to perform keyword matching on the original data obtained from the engineering overview text.

5. The method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence according to claim 1 is characterized in that: In step 2, the key information is preliminarily sorted, specifically including removing irrelevant characters and converting into a unified format.

6. The method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence according to claim 1 is characterized in that: In step 3, based on the clustering results and statistical methods, data points that deviate from the normal range in the key information that has been initially sorted are identified and marked as outliers, and then all the outliers are deleted or corrected based on other data points; Alternatively, similarity calculation is used to identify and remove duplicates, and unmatched data is considered as outliers and discarded without further identification and matching.

7. The method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence according to claim 1 is characterized in that: In step 4, a deep learning model is constructed to predict the missing data, or an existing data training model is used to predict; And according to the prediction results, a suitable strategy is selected to fill the missing data by directly using the predicted value or taking the average value or median method according to the data type.

8. The method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence according to claim 1 is characterized in that: In step 6, the data is standardized or normalized using Min-Max Scaling or Z-Score normalization neural network.

9. The method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence according to claim 1 is characterized in that: Step 7 is to infer all the missing data obtained based on the engineering indicator characteristic parameters, as well as all the data after the standardization or normalization processing, and automatically generate a new feature combination in a unified standard format.

Images