A Smart Screening Method for Compressive Strength Deterioration Data of Marine Concrete
By constructing an automated data filtering method based on machine learning algorithms, the problem of duplicate and abnormal data in marine concrete deterioration data was solved, improving data quality and management efficiency, and supporting the life assessment and model training of marine engineering projects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2026-04-01
- Publication Date
- 2026-06-30
AI Technical Summary
Existing technologies for processing marine concrete deterioration data suffer from problems such as difficulty in identifying duplicate data, difficulty in judging abnormal records, excessive human intervention, and inability to quickly process large-scale data, leading to a decline in data quality and model prediction reliability.
An automated data filtering method based on machine learning algorithms was constructed, including Excel data uploading, field reading, duplicate comparison, model prediction, error analysis, and anomaly labeling. The method uses models such as support vector machine and random forest to identify duplicate data and label abnormal data, and generates filtering logs to improve data quality.
It achieves efficient and accurate data screening, reduces manual intervention, improves data management efficiency and the credibility of model predictions, and is suitable for updating large-scale databases and assessing the lifespan of marine engineering projects.
Smart Images
Figure CN122309497A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of concrete service performance prediction and data governance technology, specifically involving a method for screening data on compressive strength degradation of marine concrete based on machine learning algorithms. Background Technology
[0002] As a core material in marine engineering structures, marine concrete is subjected to the combined effects of complex environmental factors such as chloride ion corrosion, sulfate ion corrosion, wet-dry cycles, tidal fluctuations, high humidity, and temperature changes over long periods. Its compressive strength exhibits a non-linear decline over service life. Accurately describing the degradation behavior of concrete in the marine environment requires a large amount of experimental and field monitoring data. However, due to the complex service environment of marine concrete, diverse experimental conditions, and significant differences in monitoring methods, inconsistencies often exist in the data structure, recording methods, and accuracy from different sources, significantly increasing the difficulty of data integration and screening.
[0003] In existing technologies, the screening of marine concrete deterioration data still relies primarily on manual methods, including manual data entry, manual comparison, and manual judgment. These methods are not only inefficient but also struggle to identify anomalies hidden within large datasets. For example, some data, while numerically within a reasonable range, may deviate from actual deterioration patterns due to testing interference or input errors. Furthermore, duplicate records exist between different data sources, but format differences make them difficult to identify manually. As databases continue to expand, these problems pose a serious threat to data quality.
[0004] Meanwhile, data-driven durability prediction models are becoming increasingly popular in the civil engineering field. However, machine learning models are highly sensitive to the quality of input data. If the training set contains a large amount of duplicate or anomalous data, it will lead to increased prediction bias and reduce the reliability of its engineering applications. Therefore, an intelligent screening method is needed that can automatically identify duplicate records, automatically identify anomalous records, and reduce manual intervention to ensure the data quality of the marine concrete database during the update and expansion process.
[0005] However, currently, there is no technical solution that can effectively combine Excel field reading, database comparison, machine learning prediction, and error judgment mechanisms to complete data filtering. Existing technologies also lack the ability to visualize and annotate the filtering results and generate filtering logs. Therefore, it is necessary to provide a fully automated intelligent filtering method for marine concrete compressive strength degradation data to overcome the limitations of traditional manual filtering methods. Summary of the Invention
[0006] Technical Problem: This invention aims to solve the technical problems existing in the processing of marine concrete deterioration data, such as difficulty in identifying duplicate data, difficulty in judging abnormal records, excessive human intervention, and inability to quickly process large-scale data. It proposes an intelligent screening method for marine concrete compressive strength deterioration data based on machine learning algorithms. This invention constructs a complete automated processing chain from Excel data uploading, field reading, duplicate comparison, model prediction, error analysis to anomaly labeling and result export, ensuring that the screening process is efficient, accurate, and traceable.
[0007] Technical Solution: The present invention provides an intelligent screening method for compressive strength degradation data of marine concrete, based on machine learning algorithms, comprising the following steps: Step 101: Receive the Excel data file uploaded by the user containing records of marine concrete deterioration, and read the key fields from it for data comparison and prediction. Step 102: Compare the read key field content with the corresponding fields of existing records in the database to identify duplicate data, and remove the duplicate data after identification; Step 103: Based on the error threshold set by the user, input the data after removing duplicates into the machine learning prediction model to obtain the final degradation intensity prediction value for each record. Step 104: Calculate the error based on the difference between the predicted value and the measured final compressive strength in the Excel data file; Step 105: When the error exceeds the error threshold, the corresponding record is determined to be abnormal data and marked with a preset color in the output result; Step 106: Generate and output a filtering result file containing normal data, labeled abnormal data, and filtering logs.
[0008] in: The comparison of key fields includes comparing all key fields such as initial compressive strength, water-cement ratio, chloride ion content, sulfate content, exposure age, and final compressive strength to determine whether it belongs to a duplicate record already existing in the database.
[0009] The key comparison fields also include generating combined hash values based on multiple fields to quickly identify potential duplicate records and improve comparison efficiency.
[0010] The error is the absolute or relative error between the predicted value and the measured value.
[0011] The machine learning prediction model is a support vector machine model, a random forest model, a gradient boosting model, or other models that can be used for regression prediction.
[0012] The abnormal data is visually marked in bright yellow or other preset colors in the exported Excel file for manual review.
[0013] The filtered logs include the number of duplicate data, the number of abnormal data, the error threshold, and the type of machine learning model used.
[0014] An intelligent screening system for data on compressive strength degradation of marine concrete includes: A data reading module used to receive and read user-uploaded Excel data files; This module is used to compare read fields with database records and remove duplicate data. The model prediction module is used to call the machine learning model to generate predicted values based on the error threshold. An error calculation module is used to calculate the error between predicted and measured values; An anomaly labeling module is used to identify and label abnormal data based on the magnitude of the error. This module is used to output the filtered result file and generate the filtered log.
[0015] Beneficial effects: This invention utilizes predictive models to assist in judging data quality, making the identification of abnormal records more scientific and objective; it reduces manual intervention through automated comparison mechanisms, significantly improving screening efficiency; and it enhances data verification and management efficiency through color-coding and log output mechanisms, meeting the application requirements of continuous updates for large-scale databases. This invention is of great significance for improving the quality of service performance data for marine concrete and can be extended to the screening of other material performance databases. It is applicable to the construction of concrete durability databases, service performance research, and life assessment of marine engineering structures. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the data intelligent filtering method of the present invention. Detailed Implementation
[0017] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that those skilled in the art can make various modifications or alternatives to the following embodiments without departing from the core idea of the present invention, and all such modifications or alternatives should be considered to fall within the protection scope of the present invention.
[0018] The method of this invention extracts key feature values for comparison and prediction after the Excel file is uploaded, using a field reading mechanism. The read field values are then compared with corresponding fields in the database to determine if duplicate records exist and to remove duplicate data. In the error filtering stage, the system inputs non-duplicate data into a machine learning model to generate the predicted final degradation intensity based on a user-defined error threshold. This predicted intensity is then compared with the measured intensity in the Excel file, and the magnitude of the error determines whether the data is an anomaly. If the error of a data entry exceeds the threshold, the system will highlight it in bright yellow in the final exported Excel file. Simultaneously, the system generates a filtering log to record information such as the number of duplicate entries identified, the number of anomalies, and the error threshold, enhancing the transparency and credibility of the filtering process.
[0019] Figure 1 The diagram shows the overall process of the intelligent screening method for compressive strength degradation data of marine concrete based on machine learning algorithm of the present invention. The processing order of steps 101 to 106 corresponds to the technical steps described in the specification, so as to facilitate understanding of the data processing logic of the present invention.
[0020] like Figure 1 As shown, the method of this invention can be run on servers, industrial computers, cloud processing platforms, or terminal devices with data processing capabilities. The system uses an automated process to screen marine concrete deterioration data from different sources and outputs the screening results, avoiding the inefficiency and misjudgment problems caused by manually comparing data item by item and manually judging outliers.
[0021] In step 101, the present invention reads and parses key fields from an Excel file containing user-uploaded data on the deterioration of marine concrete. The input Excel file typically originates from laboratory accelerated corrosion tests, long-term natural exposure tests, engineering service monitoring data, or literature data. The file usually contains multiple fields characterizing the concrete material and its environment, such as initial compressive strength, water-cement ratio, material admixture, chloride ion content, sulfate content, exposure age, and final compressive strength.
[0022] Because data sources may have different formats, this invention does not require uploaded files to have a fixed structure. Instead, it reads data using preset field names or field index positions. The system does not need to perform format unification or standardization on the Excel data; it only needs to extract content directly related to comparison and prediction. The read data will then be sent to subsequent data processing steps as objects to be filtered.
[0023] This step enables the system to quickly extract key variables from multi-source data provided by the user, ensuring efficient data filtering and accurate subsequent processes.
[0024] In step 102, the present invention compares the read field content with existing records in the database one by one to determine whether the data is a duplicate record. The present invention can achieve duplicate identification through multiple field comparison methods, including at least key parameters such as initial compressive strength, water-cement ratio, chloride ion content, exposure age, and final compressive strength.
[0025] In one implementation, the system compares each of the aforementioned key fields one by one. If all fields match the corresponding record in the database, the data is considered duplicate. In another implementation, to improve the efficiency of processing massive amounts of data, the system can generate a hash value based on a combination of multiple fields, and identify duplicate records through rapid comparison of these hash values.
[0026] When a data record is identified as a duplicate, the system removes it from the current filter set to prevent duplicate information from entering the model prediction stage. This not only improves the uniqueness of the database but also avoids the bias caused by duplicate data to the overall prediction model.
[0027] Therefore, this step enables the present invention to maintain efficient data cleaning capabilities even in scenarios where the database continues to expand.
[0028] In step 103, the present invention inputs the deduplicated data into the machine learning model according to the error filtering threshold set by the user, and generates the final degradation intensity prediction value for each data point.
[0029] The machine learning model may include a support vector machine regression model, a gradient boosting tree model, a random forest regression model, or other models suitable for regression prediction. The model of this invention is trained on a large number of historical samples in a database and can adapt to the strength degradation patterns of marine concrete under the coupled effects of multiple factors in the marine environment, thus enabling reliable predictions for unknown data.
[0030] The model's predicted output values serve as a crucial basis for subsequent assessments of data anomalies. This step, through a model reverse validation mechanism, ensures that the system is not only a data cleaning tool but also possesses robust predictive support capabilities.
[0031] like Figure 1 As shown, in step 104, the present invention calculates the error between the model's predicted value and the measured final compressive strength in the Excel file. The error can be absolute error, relative error, or proportional deviation, and the system can select the appropriate option according to user needs.
[0032] When the error exceeds the user-defined error threshold, the system identifies the data record as potentially abnormal data and classifies it as an object for subsequent visualization annotation.
[0033] This judgment process is based on the difference between the model output value and the measured value, avoiding the subjectivity brought about by human experience judgment, and making anomaly identification more scientific and consistent.
[0034] By controlling the error threshold, this invention can quickly identify data points that do not conform to the laws of concrete deterioration, such as abnormal records caused by experimental errors, data entry errors, or strength degradation that does not conform to the actual process.
[0035] In step 105, the present invention performs visual annotation processing on the data records identified as abnormal in step 104. Specifically, in the final exported Excel file, the entire row of abnormal records is highlighted in bright yellow, and the reason for the abnormality or error message is written in the remarks column.
[0036] This visualization method has the following advantages: (1) Make abnormal data clear and conspicuous in the output file, so that it can be quickly identified by humans; (2) Improve the efficiency of database administrators in reviewing imported data; (3) Provides an intuitive basis for filtering data during model training; (4) Enhance the transparency of the data screening process.
[0037] Therefore, visual annotation is not only a display function, but also an effective data management tool.
[0038] In step 106, the system exports the filtered dataset to a new Excel file. The exported data includes: normal data that is recommended to be retained; abnormal data marked in bright yellow; and the filtering log automatically generated by the system.
[0039] The filtered logs should include at least: the number of duplicate records, the number of abnormal records, the user-defined error threshold, the type of machine learning model used and its key parameters, the execution timestamp, and an overview of the filtering process.
[0040] The exported files can be used directly for database update operations, or for subsequent machine learning model training, ensuring the quality, stability, and reliability of the model training data.
[0041] To further illustrate the practical application effects of the present invention, a typical embodiment is given below.
[0042] Researchers screened 150 sets of corrosion test data for marine concrete. The data came from different test batches, with final compressive strength values obtained by different testers and experimental devices. The researchers uploaded the Excel files to the system of this invention.
[0043] In step 101, the system extracts key fields such as initial intensity, exposure age, water-cement ratio, and environmental ion concentration. In step 102, 21 data entries are found to be completely consistent with existing records in the database and are automatically removed. In step 103, the user sets a data error threshold of 15%, and the system generates the final predicted value of degradation intensity based on a support vector machine regression model. In step 104, the system calculates the absolute error between the predicted value and the measured value. When the error exceeds the set threshold, it is judged as abnormal data. In step 105, abnormal records are highlighted in bright yellow. Finally, in step 106, the system outputs the filtered normal and abnormal data and generates a log file containing information about the filtering process.
[0044] This embodiment demonstrates that the present invention can automatically identify duplicate and abnormal records from a large amount of heterogeneous marine concrete data from various sources, significantly improving data management efficiency, reducing the burden of manual review, and providing a high-quality data foundation for marine engineering life assessment and model training.
Claims
1. A method for intelligently screening data on the deterioration of compressive strength of marine concrete, characterized in that, This method, based on machine learning algorithms, includes the following steps: Step 101: Receive the Excel data file uploaded by the user containing records of marine concrete deterioration, and read the key fields from it for data comparison and prediction. Step 102: Compare the read key field content with the corresponding fields of existing records in the database to identify duplicate data, and remove the duplicate data after identification; Step 103: Based on the error threshold set by the user, input the data after removing duplicates into the machine learning prediction model to obtain the final degradation intensity prediction value for each record. Step 104: Calculate the error based on the difference between the predicted value and the measured final compressive strength in the Excel data file; Step 105: When the error exceeds the error threshold, the corresponding record is determined to be abnormal data and marked with a preset color in the output result; Step 106: Generate and output a filtering result file containing normal data, labeled abnormal data, and filtering logs.
2. The intelligent screening method for compressive strength degradation data of marine concrete according to claim 1, characterized in that, The comparison of key fields includes comparing all key fields such as initial compressive strength, water-cement ratio, chloride ion content, sulfate content, exposure age, and final compressive strength to determine whether it belongs to a duplicate record already existing in the database.
3. The intelligent screening method for compressive strength degradation data of marine concrete according to claim 2, characterized in that, The key comparison fields also include generating combined hash values based on multiple fields to quickly identify potential duplicate records and improve comparison efficiency.
4. The intelligent screening method for compressive strength degradation data of marine concrete according to claim 3, characterized in that, The error is the absolute or relative error between the predicted value and the measured value.
5. The intelligent screening method for compressive strength degradation data of marine concrete according to claim 4, characterized in that, The machine learning prediction model is a support vector machine model, a random forest model, a gradient boosting model, or other models that can be used for regression prediction.
6. The intelligent screening method for compressive strength degradation data of marine concrete according to claim 5, characterized in that, The abnormal data is visually marked in bright yellow or other preset colors in the exported Excel file for manual review.
7. The intelligent screening method for compressive strength degradation data of marine concrete according to claim 6, characterized in that, The filtered logs include the number of duplicate data, the number of abnormal data, the error threshold, and the type of machine learning model used.
8. The intelligent screening method for compressive strength degradation data of marine concrete according to claim 7, characterized in that, An intelligent screening system for data on compressive strength degradation of marine concrete includes: A data reading module used to receive and read user-uploaded Excel data files; This module is used to compare read fields with database records and remove duplicate data. The model prediction module is used to call the machine learning model to generate predicted values based on the error threshold. An error calculation module is used to calculate the error between predicted and measured values; An anomaly labeling module is used to identify and label abnormal data based on the magnitude of the error. This module is used to output the filtered result file and generate the filtered log.