Municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence

Abstract

The application discloses a municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence, which comprises the following steps: 1, setting engineering index characteristic parameters and specific types of each engineering index characteristic parameter; 2, obtaining key information from engineering profile text for preliminary arrangement; 3, performing clustering analysis, identifying and removing duplicate data records; 4, analyzing missing data to infer missing data; 5, matching and verifying the inferred missing data; 6, using a neural network to standardize or normalize the data; 7, integrating all the inferred missing data and all the data after standardization or normalization to form a new characteristic combination in a unified standard format; and then using a weight evaluation method of the characteristic parameters to screen out the most representative characteristic combination. The application can guarantee data quality, enhance the prediction ability and analysis value of the data, and make the process more efficient and reliable.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of road engineering cost data cleaning and preprocessing, and particularly relates to a municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence. BACKGROUND

[0002] In the field of engineering cost, data cleaning and preprocessing is a crucial first step in data analysis and machine learning. Municipal road engineering usually involves a large amount of data, including design drawings, construction records, material lists, cost accounting, etc. These data formats are diverse, sources are extensive, and contain a large amount of text, numbers and image information, increasing the difficulty of data cleaning and preprocessing.

[0003] Professional terms put forward higher requirements for data cleaning and preprocessing. Municipal road engineering data has a high update frequency and may contain missing values and outliers.

[0004] Existing technologies often rely on manual data cleaning and preprocessing, such as manual identification and extraction of key information, manual filling of missing values, etc., which is difficult to meet the processing needs of a large amount of data. Manual involvement can also affect data quality and analysis accuracy.

[0005] Therefore, how to identify and remove duplicates while ensuring data consistency or approximation, and on this basis, conduct semantic analysis, adjust the scale of data, and accumulate data as a validation dataset to verify the effectiveness of data cleaning and preprocessing has become a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0006] In view of the above defects of the prior art, the present application provides a municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence, which realizes the purpose of automatically generating new feature combinations by binding feature parameters and parameter values with source data, integrating into unified standard parameter values, while ensuring data quality, enhancing data prediction ability and analysis value, improving data processing efficiency, reducing manual intervention, making the process of data analysis and machine learning more efficient and reliable.

[0007] To achieve the above purpose, the present application discloses a municipal road engineering cost data cleaning and preprocessing method based on artificial intelligence, comprising the following steps:

[0008] Step 1, setting up engineering index feature parameters for municipal road engineering cost data that needs to be cleaned and preprocessed, and the specific type of each engineering index feature parameter;

[0009] The engineering index feature parameters include road length, road width, number of lanes, bid section name and road surface material;

[0010] Step 2, obtain the original data from the engineering profile text for keyword matching, and identify the text paragraphs related to the engineering index characteristic parameters;

[0011] and extract key information from the text paragraphs;

[0012] Then, the key information is preliminarily sorted;

[0013] Step 3, similar data deduplication and outlier detection; Specifically, using a density clustering algorithm to cluster analyze the key information that has completed preliminary sorting, identify and remove duplicate data records;

[0014] Step 4, analyze the missing data for the engineering index characteristic parameter value part of all missing engineering index characteristic parameters, specifically: infer the missing data by identifying context information or non-missing characteristic parameter values;

[0015] Step 5, matching verification of the inferred missing data;

[0016] If the matching verification is unsuccessful, steps 2 to 4 are re-executed;

[0017] If the matching verification is successful, the subsequent steps are continued;

[0018] Step 6, according to the engineering index characteristic parameters, the characteristic parameters that can be successfully matched in the engineering profile text are all standardized or normalized using neural networks;

[0019] Step 7, integrate all the missing data inferred and all the data after the standardization or normalization processing to form a new unified standard format feature combination;

[0020] Then, use the weight evaluation method of the characteristic parameters to screen out the most representative feature combination.

[0021] Preferably, the specific types of the road length, the road width and the number of lanes are all numerical values;

[0022] The specific type of the bid name is text type;

[0023] The specific type of the road surface material is enumeration type.

[0024] Preferably, the road surface material includes concrete and asphalt.

[0025] Preferably, in step 2, the natural language processing technology (NLP) is used for keyword matching of the original data obtained from the engineering profile text.

[0026] Preferably, in step 2, the preliminary sorting of the key information specifically includes removing irrelevant characters and converting it to a uniform format.

[0027] Preferably, in step 3, based on the clustering results and statistical methods, data points that deviate from the normal range in the key information that has been initially sorted are identified and marked as outliers. Then, all outliers are deleted or corrected based on other data points.

[0028] Alternatively, similarity calculations can be used to identify and remove duplicates, and unmatched data can be treated as outliers and removed without further identification and matching.

[0029] Preferably, in step 4, a deep learning model is constructed to predict the missing data, or a model is trained using existing data to make predictions.

[0030] Furthermore, based on the prediction results, an appropriate strategy is selected to fill the missing data, either by directly using the predicted value or by using the average or median method based on the data type.

[0031] Preferably, in step 6, the data is standardized or normalized using a min-max scaling or Z-score normalization neural network.

[0032] Preferably, step 7, based on the engineering indicator feature parameters, infers all the missing data obtained, and all the data after the standardization or normalization process, and automatically generates a new feature combination in a unified standard format.

[0033] The beneficial effects of this invention are:

[0034] This invention enhances the predictive power and analytical value of data while ensuring data quality, improves data processing efficiency, reduces human intervention, and makes the data analysis and machine learning process more efficient and reliable.

[0035] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0036] Figure 1 A flowchart of an embodiment of the present invention is shown. Detailed Implementation

[0037] Example

[0038] like Figure 1 As shown, the method for cleaning and preprocessing municipal road engineering cost data based on artificial intelligence includes the following steps:

[0039] Step 1, setting up engineering index characteristic parameters and specific types of each engineering index characteristic parameter for municipal road engineering cost data requiring data cleaning and preprocessing;

[0040] The engineering index characteristic parameters include road length, road width, lane number, bid section name and road surface material;

[0041] Step 1 is used to establish a characteristic parameter and parameter value type standard, and to clearly define the specific types of parameter value types. Different parameter value types determine the accuracy of the subsequent matching characteristic parameter values.

[0042] Step 1 is to ensure complete coverage of characteristic parameters and corresponding parameter value types, and to ensure consistency or approximation between engineering files, engineering profile texts and characteristic parameters and parameter values.

[0043] Step 2, obtaining original data from engineering profile texts for keyword matching to identify text passages related to engineering index characteristic parameters;

[0044] And extract key information from the text passages;

[0045] Then, the key information is preliminarily sorted out;

[0046] Step 2 is to accumulate a part of the data as a verification data set for verifying the effect of data cleaning and preprocessing.

[0047] Step 3, similar data de-duplication and outlier detection; specifically, using a density clustering algorithm to cluster analyze the key information that has been preliminarily sorted out, to identify and remove duplicate data records;

[0048] Step 4, analyzing the missing data for all missing engineering index characteristic parameter characteristic parameter values, specifically: inferring missing data by identifying context information or non-missing characteristic parameter values;

[0049] Step 5, matching and verifying the inferred missing data;

[0050] If the matching and verification is unsuccessful, steps 2 to 4 are re-executed;

[0051] If the matching and verification is successful, the subsequent steps are continued;

[0052] Step 6, using neural networks to standardize or normalize the data in the engineering profile texts that can be successfully matched according to the engineering index characteristic parameters;

[0053] Step 7, integrating all the inferred missing data and all the data that has been standardized or normalized to form a new unified standard format feature combination;

[0054] The weight evaluation method of the feature parameters is used to screen the most representative feature combination.

[0055] The feature parameters and parameter values in the actual engineering file and the engineering profile text are extracted, and the missing feature parameter values are automatically filled and supplemented according to the corresponding types of the system standard feature parameters and parameter values, so as to solve the problems in the prior art and improve the efficiency and accuracy of data cleaning and preprocessing.

[0056] The feature parameters with similar semantics are combined together, and when similar feature parameters and parameter values are identified again, the corresponding type of feature parameter value is automatically filled. Of course, the scale of the data can be adjusted according to actual needs.

[0057] In some embodiments, the specific types of road length, road width and lane number are all numerical types;

[0058] The specific type of bid name is a text type;

[0059] The specific type of pavement material is an enumeration type.

[0060] In some embodiments, the pavement material includes concrete and asphalt.

[0061] In some embodiments, in step 2, the natural language processing technology (NLP) is used to match the keywords of the original data obtained from the engineering profile text.

[0062] In some embodiments, in step 2, the preliminary arrangement of the key information specifically includes removing irrelevant characters and converting to a unified format.

[0063] In some embodiments, in step 3, according to the clustering results and statistical methods, the data points deviating from the normal range in the key information after preliminary arrangement are identified and marked as outliers, and then all the outliers are deleted or corrected based on other data points;

[0064] Alternatively, the similarity calculation is used to identify and remove duplicates, and the unmatched data is regarded as an outlier and is removed, and no longer identification matching is performed.

[0065] In some embodiments, in step 4, a deep learning model is constructed to predict the missing data, or an existing data training model is used to predict the missing data;

[0066] And according to the prediction result, a suitable strategy is selected to fill the missing data by directly using the predicted value or according to the data type to adopt the average value, median method.

[0067] Example: Engineering profile text: "Road accurate to branch", has been matched to "Road", "Branch", and from the keyword "Branch", the road level can be associated and matched, and the matching is successful.

[0068] In some embodiments, the data is standardized or normalized in step 6 using a Min-Max Scaling or Z-Score normalization neural network.

[0069] In practical applications, the neural network can learn the distribution characteristics of the data, form memory points, and quickly process similar data in the future. Example: Engineering profile text: "Road accurate to branch", "Road accurate to" is a set of feature parameters in the current engineering profile text, "branch" is the feature value, as well as similar arrangement and text description.

[0070] In some embodiments, step 7 automatically generates a new unified standard format feature combination based on the engineering index feature parameters, all missing data obtained, and all data after standardization or normalization.

[0071] In practical applications, the above technical means can enhance the prediction ability and analysis value of the data.

[0072] The above describes the preferred embodiments of the present application. It should be understood that those skilled in the art can make many modifications and changes without creative labor based on the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiment based on the existing technology according to the concept of the present application shall be within the protection scope determined by the claims.

Claims

1. An artificial intelligence-based municipal road engineering cost data cleaning and preprocessing method, characterized in that, The method comprises the following steps: Step 1, setting up engineering index characteristic parameters for municipal road engineering cost data in need of data cleaning and preprocessing, and specific types of each engineering index characteristic parameter; The engineering index characteristic parameters include road length, road width, lane number, bid section name and road surface material; Step 2, obtaining original data from engineering profile text for keyword matching, identifying text paragraphs related to the engineering index characteristic parameters; and extracting key information from the text paragraphs; then preliminarily arranging the key information; Step 3, similar data de-duplication and outlier detection; specifically, using a density clustering algorithm to perform clustering analysis on the key information after preliminary arrangement, identifying and removing duplicate data records; Step 4, analyzing missing data for the characteristic parameter value part of all missing engineering index characteristic parameters; specifically, inferring the missing data by identifying context information or non-missing characteristic parameter values; Step 5, matching and verifying the inferred missing data; if the matching and verifying is unsuccessful, re-executing steps 2 to 4; if the matching and verifying is successful, then continuing to execute subsequent steps; Step 6, using neural networks to standardize or normalize data for the characteristic parameters that can be successfully matched in the engineering profile text according to the engineering index characteristic parameters; Step 7, integrating all the inferred missing data and all the data after the standardization or normalization to form a new characteristic combination in a unified standard format; then using a characteristic parameter weight evaluation method to select the most representative characteristic combination.

2. The method for municipal road engineering cost data cleaning and preprocessing based on artificial intelligence according to claim 1, characterized in that, The specific types of the road length, the road width and the lane number are all numerical types; the specific type of the bid section name is a text type; the specific type of the road surface material is an enumeration type.

3. The method for municipal road engineering cost data cleaning and preprocessing based on artificial intelligence according to claim 1, characterized in that, The road surface material includes concrete and asphalt.

4. The method for municipal road engineering cost data cleaning and preprocessing based on artificial intelligence according to claim 1, characterized in that, In step 2, natural language processing technology (NLP) is used to perform keyword matching on the original data obtained from the engineering profile text. 5.The method of claim 1, wherein the method further comprises: determining whether the data is missing; and if the data is missing, determining whether the missing data is a missing value or a null value. In step 2, preliminarily arranging the key information specifically includes removing irrelevant characters and converting to a unified format. 6.The method of claim 1, wherein the method further comprises: In step 3, according to the clustering results and statistical methods, identifying data points deviating from the normal range in the key information after preliminary arrangement and marking them as outliers, then deleting or correcting all the outliers based on other data points; ​ or, using similarity calculation to identify and remove duplicates, and regarding unmatched data as outliers and removing them without further identification and matching. 7.The method of claim 1, wherein the method further comprises: determining whether the data is missing; and if the data is missing, determining whether the missing data is a missing value or a null value. In step 4, constructing a deep learning model to predict the missing data, or using an existing data training model to predict; and according to the prediction results, selecting appropriate strategies to fill in the missing data by directly using the predicted values or using average value or median value methods according to the data type.

8. The method for municipal road engineering cost data cleaning and preprocessing based on artificial intelligence according to claim 1, characterized in that, In step 6, using Min-Max Scaling or Z-Score standardization neural networks to standardize or normalize data. 9.The method of claim 1, wherein the method further comprises: Step 7: Based on the engineering index characteristic parameters, all the missing data obtained, and all the data after the standardization or normalization processing, a new unified standard format characteristic combination is automatically generated. ​

Images