Automatic data set cleaning method and system for large model in geotechnical engineering field

Abstract

The invention discloses an automatic data set cleaning method and system for a large model in the geotechnical engineering field, and belongs to the technical field of data processing and artificial intelligence. The method comprises the following steps: acquiring multiple source files under a specified directory, reading multiple worksheet contents, and analyzing to generate an analyzed data table; performing structured verification and optimization on the analyzed data table, eliminating invalid rows and columns and recording dimension changes; performing multi-level standardization processing on fields in the data table, and generating a field mapping relation table; performing anomaly detection on the standardized fields; and outputting the standardized data set and the whole-process audit log. According to the method, the defects that traditional manual cleaning is low in efficiency and poor in consistency are overcome, high-quality and standardized structured data support can be provided for large model training in the geotechnical engineering field, and the efficiency and reliability of data preparation are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of data processing and artificial intelligence, and in particular to an automated method and system for cleaning datasets of large models in the field of geotechnical engineering. Background Technology

[0002] Data in geotechnical engineering is characterized by its multi-source nature, heterogeneity, and high noise levels. Data sources include laboratory experiments, in-situ tests, engineering monitoring, and geological survey reports, and formats encompass structured tables, semi-structured reports, and unstructured text. Before being used to train large-scale models, this data must undergo rigorous cleaning and standardization; otherwise, it will lead to decreased model performance, distorted predictions, and even risks in engineering decision-making.

[0003] Currently, common data cleaning methods suffer from the following drawbacks: 1. Lack of domain specificity: General cleaning algorithms (such as simple threshold filtering or interpolation) cannot effectively identify anomalous patterns in geotechnical engineering data that contain professional knowledge (such as illogical combinations of geotechnical parameters). 2. Low degree of automation: Existing methods largely rely on manual rules and experience-based judgments, resulting in cumbersome processes, low efficiency, and difficulty in meeting the needs of large-scale data preprocessing. 3. Lack of process traceability: Decisions and changes during the cleaning process lack complete audit logs, leading to unclear data lineage and poor reliability and reproducibility of results. 4. Lack of quality monitoring: The lack of systematic quality monitoring and quantitative evaluation of the entire data cleaning process makes it impossible to monitor the health status of the data in real time.

[0004] Therefore, a data cleaning solution is needed that can deeply integrate geotechnical engineering business rules, achieve a high degree of automation, and have full-process quality monitoring capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide an automated cleaning method and system for large models in the field of geotechnical engineering, so as to achieve efficient, accurate, traceable automated cleaning and quality control of multi-source heterogeneous geotechnical engineering data.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] This invention provides an automated dataset cleaning method for large models in the field of geotechnical engineering, including:

[0008] The system retrieves multiple source files from a specified directory, reads the contents of multiple worksheets, and parses them to generate a data table. The multiple source files cover project information, field exploration borehole information, borehole stratification, standard penetration test data, double-bridge static penetration test data, sampling data, geotechnical test data, pile foundation calculations, and standard layer data.

[0009] Perform structured validation and optimization on the parsed data table, eliminate invalid rows and columns, record dimension changes, and remove abnormal data;

[0010] The fields in the data table are standardized at multiple levels, and a field mapping table is generated; the standardization process includes naming standardization and format standardization.

[0011] Perform anomaly detection on the standardized fields and remove abnormal data;

[0012] Output a standardized dataset after removing outlier data and a full-process audit log.

[0013] Preferably, the step of obtaining multiple source files in a specified directory, reading the contents of multiple worksheets, and parsing to generate a data table includes:

[0014] Each worksheet is parsed to generate a metadata summary, including the table structure fingerprint, data dimensions, and data type distribution. All parsing operations are recorded in the data table in ISO 8601 timestamp format.

[0015] When encountering encrypted or corrupted files, the file is automatically skipped and a high-level alert is recorded in the data table, while other file processing tasks continue to be performed.

[0016] Preferably, the dimensional change is calculated as follows: , in, Indicates the rate of change of dimension. and These represent the number of rows in the data table before and after cleaning, respectively. When the dimensional change rate exceeds a preset threshold of 20%, an anomaly flag is triggered, abnormal data is removed, and an alarm message containing a detailed dimensional change report is sent.

[0017] Preferably, the naming standardization refers to using regular expressions to remove extra spaces in field names and converting them to underscore format;

[0018] The standardization of the format refers to forcibly converting the date field into a unified format according to the year-month-day format; standardizing the units of mechanical parameters; and uniformly converting the intensity units into megapascals.

[0019] The generated field mapping table is represented as a matrix. ,in The number of fields is represented by the matrix columns, which record the original field names, standardized field names, and transformation operation types, respectively.

[0020] Preferably, the anomaly detection of the standardized fields includes:

[0021] The fields are standardized and preprocessed to eliminate dimensions and form data points;

[0022] The optimal neighborhood parameter k is determined using the elbow rule;

[0023] Calculate the local reachability density of each data point based on the k-distance neighborhood;

[0024] Calculate the local outlier factor based on the local reachability density of data points and their k-distance neighborhood data points;

[0025] Abnormal data points are identified based on local outlier factors and dynamic thresholds, and the anomaly detection results are recorded.

[0026] Preferably, the standardization preprocessing of the fields includes:

[0027] Perform Z-score standardization on numeric fields;

[0028] Perform one-hot encoding conversion on categorical fields;

[0029] Check data integrity, handle missing values, and generate a data quality assessment report.

[0030] Preferably, the locally reachable density is calculated as follows: , in, Representing data points Locally achievable density, For data points and data points The reachable distance, Representing data points The set of k-distance neighborhood data points; The local outlier factor is calculated as follows: , in, Representing data points Local outlier Representing data points The locally achievable density.

[0031] Preferably, the step of identifying anomalous data points based on local outlier factors and dynamic thresholds, and recording the anomaly detection results, includes:

[0032] Calculate the interquartile range of the local outlier for each data point. : ; It is the third quartile. It is the first quartile;

[0033] Set the dynamic threshold to ;

[0034] If the local outlier factor of a data point exceeds the dynamic threshold, the corresponding data point is marked as an outlier and recorded in the outlier registry.

[0035] This invention also provides an automated dataset cleaning system for large models in the geotechnical engineering field, used to implement the aforementioned automated dataset cleaning method for large models in the geotechnical engineering field. The system includes:

[0036] The data acquisition and parsing module is used to acquire multiple source files in a specified directory, read the contents of multiple worksheets and parse them to generate data tables; the multiple source files cover project information, field exploration borehole information, borehole stratification, standard penetration test data, double-bridge static penetration test, sampling data, geotechnical test data, pile foundation calculation and standard layer data;

[0037] The structured validation module is used to perform structured validation and optimization on the parsed data table, eliminate invalid rows and columns, record dimension changes, and remove abnormal data.

[0038] The standardization module is used to perform multi-level standardization on fields in the data table and generate a field mapping table; the standardization process includes naming standardization and format standardization.

[0039] The anomaly detection module is used to detect anomalies in the standardized fields and remove abnormal data.

[0040] The output module is used to output a standardized dataset with outlier data removed and a full-process audit log.

[0041] Preferably, the system further includes:

[0042] The quality monitoring module monitors changes in data quality in real time.

[0043] The early warning notification module automatically sends abnormal warning information;

[0044] The audit traceability module records the entire anomaly detection process;

[0045] The interface service module provides standardized data access interfaces.

[0046] The beneficial effects achieved by this invention are as follows:

[0047] This invention achieves batch processing and intelligent cleaning of heterogeneous multi-source data in geotechnical engineering by constructing an automated pipeline for multi-source data parsing, structured verification, field standardization, outlier detection, and end-to-end quality control. This invention features fault tolerance mechanisms and mandatory field verification, automatically standardizing data structures and field formats, identifying and handling outliers based on statistical methods, and ensuring the traceability of the data processing process. This invention overcomes the shortcomings of traditional manual cleaning, such as low efficiency and poor consistency, and provides high-quality, standardized structured data support for training large models in the geotechnical engineering field, significantly improving the efficiency and reliability of data preparation. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the automated dataset cleaning method for large models in the geotechnical engineering field provided by the present invention.

[0049] Figure 2 This is a diagram showing the abnormal detection results of the dual-bridge static probe project in an embodiment of the present invention;

[0050] Figure 3 These are anomaly detection result data tables with different table names in embodiments of the present invention;

[0051] Figure 4 This is a quality score chart for each data table in an embodiment of the present invention;

[0052] Figure 5 This is a graph showing the relationship between the number of samples and the number of outliers in an embodiment of the present invention;

[0053] Figure 6 This is a sorting chart of the abnormal proportions of each data table in an embodiment of the present invention;

[0054] Figure 7 This is a graph showing the maximum LOF values ​​of each table in this embodiment of the invention;

[0055] Figure 8 This is a pie chart showing the distribution of abnormal points in an embodiment of the present invention;

[0056] Figure 9 This is a comparison chart of the anomaly detection results of each data table in the embodiments of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0058] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0059] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0060] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0061] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0062] It should be emphasized here that the step markers mentioned below are not a limitation on the order of the steps, but should be understood as meaning that the steps can be executed in the order mentioned in the embodiments, or in a different order than in the embodiments, or several steps can be executed simultaneously.

[0063] This invention provides an automated dataset cleaning method for large models in the field of geotechnical engineering. (See also...) Figure 1 This includes the following steps:

[0064] S1: Obtain multiple source files in the specified directory, read the contents of multiple worksheets, and parse them to generate a data table;

[0065] In step S1, reading and parsing the contents of multiple worksheets includes: parsing each worksheet, generating a metadata summary, including the table structure fingerprint (MD5 checksum), data dimensions (rows × columns) and data type distribution, and recording all parsing operations in the data table in ISO 8601 timestamp format, which is called the first parsing log.

[0066] S2 performs structured validation and optimization on the parsed data table, automatically eliminates invalid rows and columns and records dimension changes. When the dimension changes exceed the preset threshold, an anomaly flag is triggered and abnormal data is removed.

[0067] S3 performs multi-level standardization on the fields in the data table, unifies naming rules and data format, and generates a field mapping relationship table;

[0068] Step S3 includes a naming standardization stage and a format standardization stage. After standardization, a field mapping matrix is ​​generated. ,in The matrix columns record the original field names, standardized field names, and transformation operation types, respectively, to provide a complete traceability chain for quality auditing.

[0069] S4. Identify outliers in the field based on the local outlier factor algorithm, perform anomaly detection through density comparison method, and record anomaly handling logs.

[0070] In step S4, anomaly detection specifically includes the following sub-steps:

[0071] S41. Standardize and preprocess the multimodal data in the field to eliminate the influence of units;

[0072] S42. Configure local neighborhood parameters based on data distribution characteristics, and determine the optimal neighborhood parameters;

[0073] S43. Based on the optimal neighborhood parameters determined above, calculate the local reachability density of each data point based on the k-distance neighborhood.

[0074] S44. Calculate the local outlier factor by comparing the local reachability density of a data point with its neighboring points;

[0075] S45. Identify outlier data points based on local outlier factors and dynamic thresholds, and remove outlier data.

[0076] Furthermore, in step S41, the standardization preprocessing procedure is as follows:

[0077] Perform Z-score standardization on numeric fields; perform one-hot encoding conversion on categorical fields; check data integrity and handle missing values; generate a data quality assessment report.

[0078] Furthermore, in step S42, local neighborhood parameters are configured based on the data distribution characteristics, as follows:

[0079] Analyze the data distribution characteristics to determine the initial range of k values; optimize the selection of k values ​​using the elbow rule; verify the impact of k values ​​on the algorithm's sensitivity; and determine the optimal k value parameters.

[0080] Furthermore, in step S43, the local reachability density of each data point is calculated based on the k-distance neighborhood, as follows:

[0081] Calculate the k-distance for each data point; determine the k-distance neighborhood for each data point; calculate the reachability distance between points within the neighborhood; calculate the local reachability density based on the reachability distance.

[0082] Furthermore, in step S44, the local outlier factor is calculated, as follows:

[0083] Calculate the local reachability density ratio of each point in the neighborhood; calculate the local outlier based on the local reachability density ratio; normalize the local outlier; and establish a local outlier distribution map based on the normalized local outlier.

[0084] S5 performs mandatory validation on core key fields in accordance with geotechnical engineering business rules, calculates missing rate and anomaly rate indicators, and implements full-process quality monitoring.

[0085] S6 outputs a fully structured standardized dataset and a complete audit log, and performs integrity checks on the output results.

[0086] It should be noted that the method of the present invention can be used in combination with other anomaly detection algorithms in the geotechnical engineering data cleaning process to form a multi-level anomaly detection system.

[0087] It should be noted that the method of the present invention supports real-time data stream processing and can dynamically update the anomaly detection model.

[0088] The following is a detailed description of the automated dataset cleaning method for large models in the geotechnical engineering field provided by the present invention, using a specific implementation case. The technical implementation is based on the Python 3.10 environment, and its core dependencies are scientific computing libraries such as pandas 1.5.0, numpy 1.23.0, and scikit-learn 1.2.0.

[0089] An automated dataset cleaning method for large models in the geotechnical engineering field is proposed, and the specific implementation process is as follows:

[0090] Step S1: Obtain multiple source files in the specified directory, read the contents of multiple worksheets and parse them;

[0091] Specifically, the system receives data cleaning task instructions through an interface and automatically accesses a pre-configured storage server. The system uses the `pd.ExcelFile()` method to parse multi-worksheet Excel documents, and `pd.read_csv(encoding='utf-8')` to read CSV files. It also supports TSV format text files exported from a PostgreSQL database. During parsing, the system generates a metadata digest for each worksheet, including the table structure fingerprint (MD5 checksum), data dimensions (rows × columns), and data type distribution. All parsing operations are recorded in the data table using ISO 8601 timestamp format. When encountering encrypted or corrupted files, the system automatically skips the file and records a high-level alert (ERROR_LEVEL=3) in the data table, while continuing to execute other file processing tasks.

[0092] In this embodiment, the cleaning process covers nine data tables, including project information, field exploration borehole information, borehole stratification, standard penetration test data, double-bridge static penetration test data, sampling data, geotechnical test data, pile foundation calculation and standard layer data. The overall data integrity is good, with only a slight missing rate of 0.86% in the geotechnical test data.

[0093] Step S2: Perform structured validation and optimization on the parsed data table, automatically eliminate invalid rows and columns and record dimension changes. When the changes exceed the preset threshold, trigger an anomaly flag.

[0094] Specifically, this phase employs a statistically based structured analysis method. The system first uses the `df.isnull().all(axis=1)` algorithm to identify completely empty rows, and `df.isnull().all(axis=0)` to identify completely empty columns. Then, it executes `df.dropna(how='all', axis=0)` and `df.dropna(how='all', axis=1)` to perform batch removal. The system calculates the rate of change of data table dimensions in real time. :

[0095] ,

[0096] in and These represent the number of rows in the data table before and after cleaning, respectively. When the dimensional change rate exceeds a preset threshold of 20%, the system triggers an exception flag with STATUS_CODE=WARNING and sends an alert email containing a detailed dimensional change report to the system administrator.

[0097] Step S3: Perform multi-level standardization on the fields in the data table, unify naming rules and data format, and generate a field mapping relationship table;

[0098] Specifically, the system loads a predefined geotechnical engineering ontology library, which contains 127 standard field name mapping rules.

[0099] During the naming standardization phase, the system uses the regular expression re.sub(r'\s+', '_', str(col).lower().strip()) to remove extra spaces in field names and convert them to underscore format.

[0100] During the format standardization phase, date fields were forcibly converted into a unified format according to the year-month-day format, and the units of mechanical parameters were standardized. The intensity units were uniformly converted to the megapascal (MPa) dimension using the formula df[col] = df[col] * conversion_factor.

[0101] The system generates a field mapping matrix. ,in The matrix columns record the original field names, standardized field names, and transformation operation types, respectively, to provide a complete traceability chain for quality auditing.

[0102] Step S4: Identify outliers in the data based on the Local Outlier Factor (LOF) algorithm, perform anomaly detection using the density comparison method, and record the anomaly handling log;

[0103] Specifically, this stage employs an improved LOF algorithm for density clustering anomaly detection.

[0104] Step 4.1 Perform multimodal data preprocessing:

[0105] Apply Z-score normalization to numeric fields:

[0106] ,

[0107] in, The standardized value. The original data, The characteristic mean, The standard deviation is denoted as .

[0108] One-hot encoding is applied to categorical fields to generate sparse matrices. ,in For the sample size, This represents the number of categories.

[0109] Step 4.2 Optimize the LOF parameters using the elbow rule to determine the optimal neighborhood parameters;

[0110] The system in Calculate different values ​​within the range [10, 50] The profile coefficient corresponding to the value is selected based on the largest profile coefficient greater than 0.6. The value is used as the optimal neighborhood parameter.

[0111] Step 4.3 Calculate the local reachability density of each data point based on the k-distance neighborhood. The local reachability density represents the derivative of the average reachability distance between the data point and its neighboring data points, measuring the distance relationship between the data point and other data points. The calculation is as follows:

[0112] ,

[0113] in, Representing data points Locally achievable density, achievable distance , Representing data points and Euclidean distance, Representing data points k-nearest neighbor distance, Representing data points The set of k-distance neighborhood data points.

[0114] Step 4.4 Calculate the local outlier factor based on the local reachability density, as follows:

[0115] ,

[0116] in, Representing data points Local outlier Representing data points The local achievable density.

[0117] Step 4.5 employs a dynamic threshold identification mechanism to identify abnormal data points, specifically as follows:

[0118] Calculate the interquartile range of the local outlier for each data point. Set the dynamic threshold to All data points whose local outlier exceeds this dynamic threshold are marked as outliers and recorded in the outlier registry. It is the third quartile. It is the first quartile.

[0119] In this embodiment, the Local Outlier Factor (LOF) algorithm was used to analyze eight data tables: field exploration borehole information, borehole stratification, standard penetration test (SPT) data, double-bridge static penetration test data, sampling data, geotechnical test data, pile foundation calculations, and standard layer data. The resulting visualization includes seven types of charts related to data quality analysis, each providing support for data quality control from different dimensions. Figure 2The visualization is a comprehensive 2×2 layout analysis chart designed to systematically display anomaly detection results based on the Local Outlier Factor (LOF) algorithm. The top left corner shows the spatial distribution of outliers, using a scatter plot to present the data distribution in the two-dimensional space formed by the first two numerical features. Blue dots indicate normal samples, while red triangles mark anomalous samples identified by the algorithm, clearly distinguished by a legend. The top right corner shows a histogram of LOF values, revealing the statistical characteristics of local outliers for all samples in a frequency distribution format. Sky blue bars visually represent the number of samples in different LOF intervals, with red dashed lines clearly marking the preset anomaly thresholds. The bottom left corner shows a LOF value ranking chart, arranging all samples in descending order of their LOF values ​​and presenting them as a bar chart. The bars are distinguished by whether they exceed the threshold, using red (abnormal) and light coral (normal) colors respectively. Blue horizontal dashed lines further reinforce the threshold lines, clearly showing the ranking and distribution of outlier levels. The bottom right corner shows an anomaly detection statistics bar chart. Figure 6 A bar chart sorted by the anomaly rate of each data table. The horizontal axis represents the data table name, and the vertical axis represents the anomaly rate (unit: %). It shows that the anomaly rate of the sampled data is 30.43%, the anomaly rate of the field exploration borehole information is 25.58%, and the anomaly rate of the remaining data tables decreases in turn. It can intuitively present the differences in the anomaly rate of each table to screen key objects for quality control. Figure 5 This is a scatter plot showing the relationship between the number of samples and the number of outliers. The horizontal axis represents the total number of samples, and the vertical axis represents the number of outliers. The color depth of the scatter plot is positively correlated with the proportion of outliers. By analyzing the distribution of scatter plots corresponding to the sampled data (total samples 207, outliers 63) and the field exploration borehole information (total samples 172, outliers 44), the correlation between the total number of samples, the number of outliers, and the proportion of outliers is clearly shown, which facilitates the identification of densely populated outlier data tables. Figure 4 The bar chart represents the quality scores of each data table. The horizontal axis represents the data table name, and the vertical axis represents the quality score (out of 100) determined based on the calculation ability of the anomaly ratio and the local outlier factor (LOF). It shows that the pile foundation calculation and standard layer data scores are 100, the double bridge static exploration score is 99.4, and the scores of the sampling data and field exploration borehole information are relatively low. The quality level of each table can be quantitatively presented to support the classification of quality grades. Figure 3 This table summarizes the detailed results of LOF anomaly detection, with columns including table name, total number of samples, number of outliers, anomaly ratio, and maximum LOF. It clarifies the quantitative indicators for each table, with the highest anomaly ratio (30.43%) in the sampled data, providing data support for in-depth anomaly analysis. The maximum LOF of some tables is marked "N / A", which is initially judged to be related to insufficient feature dimensions or abnormal numerical distribution after data preprocessing, resulting in the algorithm's inability to effectively calculate local density. Figure 9A dual-series bar chart was used to compare the anomaly detection results of various data tables. The horizontal axis represents the data table name, and the vertical axis represents the number or percentage of data. The red bars represent the number of anomalies, and the blue bars represent the anomaly percentage (%). The chart shows that the two indicators of the sampled data (63 anomalies, 30.4% percentage) and the field exploration borehole information (44 anomalies, 25.6% percentage) are both at a high level, while the indicators of the other tables are close to zero. The two indicators can be compared simultaneously to identify the core focus of the detection. Figure 8 The pie chart shows the distribution of outliers, with each sector corresponding to a different data table. The sector percentage represents the proportion of outliers in the corresponding table to the total outliers. The data shows that the sampled data accounts for 42.9% and the field exploration borehole information accounts for 29.9%, with the two combined exceeding 70%. This visually presents the distribution percentage of outliers to clearly identify the main sources. Figure 7 The bar chart shows the maximum LOF value (valid data) for each table. The horizontal axis represents the table name, and the vertical axis represents the maximum LOF value. The red dashed line in the chart marks the anomaly threshold of 2.0. It shows that the maximum LOF value of the double-bridge static probe is 5.12 (above the threshold), and the LOF values ​​of the pile foundation calculation and standard layer data are 1.06 and 1.61 respectively (both below the threshold). This can determine the degree of local anomalies in each table to identify obviously abnormal data tables.

[0120] Step S5: Output and completeness verification of results.

[0121] Specifically, the system outputs a standardized dataset to generate a quality assessment report conforming to the ISO 25012 standard. Audit logs are recorded in JSON format, providing statistical explanations of the classification results using tables, images, and text.

[0122] Based on the same inventive concept, this invention also provides an automated dataset cleaning system for large models in the field of geotechnical engineering, comprising:

[0123] The data acquisition and parsing module is used to acquire multiple source files in a specified directory, read the contents of multiple worksheets, and parse them to generate data tables. The multiple source files cover project information, field exploration borehole information, borehole stratification, standard penetration test data, double-bridge static penetration test data, sampling data, geotechnical test data, pile foundation calculations, and standard layer data.

[0124] The structured validation module is used to perform structured validation and optimization on the parsed data table, eliminate invalid rows and columns, record dimension changes, and remove abnormal data.

[0125] The standardization module is used to perform multi-level standardization on fields in the data table and generate a field mapping table; the standardization process includes naming standardization and format standardization.

[0126] The anomaly detection module is used to detect anomalies in the standardized fields and remove abnormal data.

[0127] The output module is used to output a standardized dataset with outlier data removed and a full-process audit log.

[0128] Furthermore, the system also includes:

[0129] The quality monitoring module monitors changes in data quality in real time.

[0130] The early warning notification module automatically sends abnormal warning information;

[0131] The audit traceability module records the entire anomaly detection process;

[0132] The interface service module provides standardized data access interfaces.

[0133] It is worth noting that the system embodiment corresponds to the above method embodiment. The implementation methods of the above method embodiments are all applicable to the system embodiment and can achieve the same or similar technical effects, so they will not be described in detail here.

[0134] Based on the same inventive concept, the present invention also provides a computer-readable storage medium for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the above-described automated dataset cleaning method for large models in the field of geotechnical engineering.

[0135] Based on the same inventive concept, the present invention also provides a computing device comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the above-described automated dataset cleaning method for large models in the field of geotechnical engineering.

[0136] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An automated dataset cleaning method for large models in geotechnical engineering, characterized in that, include: Retrieve multiple source files in a specified directory, read the contents of multiple worksheets, and parse them to generate a data table; The multi-source files cover project information, field exploration borehole information, borehole stratification, standard penetration test data, double-bridge static penetration test, sampling data, geotechnical test data, pile foundation calculations, and standard layer data; Perform structured validation and optimization on the parsed data table, eliminate invalid rows and columns, record dimension changes, and remove abnormal data; The fields in the data table are standardized at multiple levels, and a field mapping table is generated; the standardization process includes naming standardization and format standardization. Perform anomaly detection on the standardized fields and remove abnormal data; Output a standardized dataset after removing outlier data and a full-process audit log.

2. The automated dataset cleaning method for large models in geotechnical engineering as described in claim 1, characterized in that, The process of obtaining multiple source files in a specified directory, reading the contents of multiple worksheets, and parsing them to generate a data table includes: Each worksheet is parsed to generate a metadata summary, including the table structure fingerprint, data dimensions, and data type distribution. All parsing operations are recorded in the data table in ISO 8601 timestamp format. When encountering encrypted or corrupted files, the file is automatically skipped and a high-level alert is recorded in the data table, while other file processing tasks continue to be performed.

3. The automated dataset cleaning method for large models in geotechnical engineering as described in claim 1, characterized in that, The dimensional change is calculated as follows: ， in, Indicates the rate of change of dimension. and These represent the number of rows in the data table before and after cleaning, respectively. When the dimensional change rate exceeds a preset threshold of 20%, an anomaly flag is triggered, abnormal data is removed, and an alarm message containing a detailed dimensional change report is sent.

4. The automated dataset cleaning method for large models in geotechnical engineering as described in claim 1, characterized in that, The naming standardization refers to using regular expressions to remove extra spaces in field names and convert them to underscore format. The standardization of the format refers to forcibly converting the date field into a unified format according to the year-month-day format; standardizing the units of mechanical parameters; and uniformly converting the intensity units into megapascals. The generated field mapping table is represented as a matrix. ,in The number of fields is represented by the matrix columns, which record the original field names, standardized field names, and transformation operation types, respectively.

5. The automated dataset cleaning method for large models in geotechnical engineering as described in claim 1, characterized in that, The anomaly detection of the standardized fields includes: The fields are standardized and preprocessed to eliminate dimensions and form data points; The optimal neighborhood parameter k is determined using the elbow rule; Calculate the local reachability density of each data point based on the k-distance neighborhood; Calculate the local outlier factor based on the local reachability density of data points and their k-distance neighborhood data points; Abnormal data points are identified based on local outlier factors and dynamic thresholds, and the anomaly detection results are recorded.

6. The automated dataset cleaning method for large models in geotechnical engineering as described in claim 5, characterized in that, The standardization preprocessing of the fields includes: Perform Z-score standardization on numeric fields; Perform one-hot encoding conversion on categorical fields; Check data integrity, handle missing values, and generate a data quality assessment report.

7. The automated dataset cleaning method for large models in geotechnical engineering as described in claim 5, characterized in that, The locally accessible density is calculated as follows: ， in, Representing data points Locally achievable density, For data points and data points The reachable distance, Representing data points The set of k-distance neighborhood data points; The local outlier factor is calculated as follows: ， in, Representing data points Local outlier Representing data points The locally achievable density.

8. The automated dataset cleaning method for large models in geotechnical engineering as described in claim 7, characterized in that, The process of identifying anomalous data points based on local outlier factors and dynamic thresholds, and recording the anomaly detection results, includes: Calculate the interquartile range of the local outlier for each data point. : ; It is the third quartile. It is the first quartile; Set the dynamic threshold to ; If the local outlier factor of a data point exceeds the dynamic threshold, the corresponding data point is marked as an outlier and recorded in the outlier registry.

9. An automated dataset cleaning system for large models in geotechnical engineering, characterized in that, The system for implementing the automated dataset cleaning method for large models in the geotechnical engineering field as described in claim 1 includes: The data acquisition and parsing module is used to acquire multiple source files in a specified directory, read the contents of multiple worksheets and parse them to generate data tables; the multiple source files cover project information, field exploration borehole information, borehole stratification, standard penetration test data, double-bridge static penetration test, sampling data, geotechnical test data, pile foundation calculation and standard layer data; The structured validation module is used to perform structured validation and optimization on the parsed data table, eliminate invalid rows and columns, record dimension changes, and remove abnormal data. The standardization module is used to perform multi-level standardization on fields in the data table and generate a field mapping table; the standardization process includes naming standardization and format standardization. The anomaly detection module is used to detect anomalies in the standardized fields and remove abnormal data. The output module is used to output a standardized dataset with outlier data removed and a full-process audit log.

10. The automated dataset cleaning system for large models in geotechnical engineering as described in claim 9, characterized in that, The system also includes: The quality monitoring module monitors changes in data quality in real time. The early warning notification module automatically sends abnormal warning information; The audit traceability module records the entire anomaly detection process; The interface service module provides standardized data access interfaces.

Images