CPT file format recognition and analysis method and system based on heuristic rules

Abstract

This invention discloses a heuristic rule-based method and system for CPT file format recognition and parsing. The method employs a three-level priority progressive format detection approach: the first level detects the full-text marker character pattern to determine the delimiter marker format; the second level detects the blank line position pattern to determine the DAT blank line format; and the third level uses first-line lexical feature analysis to distinguish between standard column formats with and without headers. It also includes adaptive processing mechanisms such as automatic pre-completion of non-standard depth values ​​and automatic alignment of delimiter marker interval data. This invention achieves zero-configuration automatic import of CPT data without manual intervention, supports more than six mainstream domestic instrument formats, and improves the data parsing success rate to over 95%, significantly enhancing the efficiency and reliability of geotechnical engineering investigation data processing.

Description

Technical Field

[0001] This invention relates to the field of geotechnical engineering investigation data processing technology, and in particular to a method and system for automatic identification and parsing of multi-source static cone penetration test (CPT) data file formats based on heuristic rules. It is applicable to zero-configuration batch import and automatic parsing of heterogeneous data files output by static cone penetration test instruments of different manufacturers and models. Background Technology

[0002] Static cone penetration testing (CPT) is one of the core in-situ testing methods in geotechnical engineering investigation. It involves pressing a cone-shaped probe into the soil layer at a standard rate and continuously measuring the cone resistance (qc) and sidewall friction (fs), providing crucial data for stratigraphic division and soil type identification. After decades of development, the domestic geotechnical engineering investigation industry has formed a system of static cone penetration testing instruments and equipment from dozens of different manufacturers, including various models from companies such as Huakan, Qingan, and the Geotechnical Survey Instrument Factory.

[0003] Each CPT instrument outputs data files in a different format. Currently, the industry-standard formats include: single-column numerical format (containing only specific penetration resistance data), two-column format (paired depth and cone resistance), three-column format (three columns of data: depth, cone resistance, and sidewall friction), separator-marked format (using specific characters to mark data ranges for each parameter), and standard DAT format (containing multiple lines of metadata and blank lines separating data areas). Different manufacturers also exhibit significant differences in header structure, metadata location, and depth value encoding methods within the same format type.

[0004] Existing commercial software such as CPTask and GEO5 typically employ one of the following methods when importing CPT data: (1) requiring users to pre-specify the file format type; (2) supporting only a few international standard formats such as GEF (Geotechnical Exchange Format) / GME; (3) providing format templates for users to manually configure field mapping relationships. The above-mentioned existing technologies have the following problems in practical applications:

[0005] (i) Limited format compatibility: Commercial software typically only supports a limited number of international standard formats (GEF / GME), and lacks compatibility with the non-standard output formats of a large number of small and medium-sized domestic manufacturers; users need to manually convert files to software-supported formats before importing data, which increases the workload of preprocessing and is prone to human error during the conversion process.

[0006] (ii) Reliance on manual prediction and configuration: Users must understand the file format type in advance and manually select the corresponding parsing template. When batch processing CPT files from different instrument manufacturers and different project periods, confirming the format type one by one is extremely time-consuming (average time of 2-5 minutes per file), and is prone to data parsing errors or data loss due to misjudgment.

[0007] (iii) Lack of abnormal format handling mechanism: Some vendors' data files have non-standard expression methods, such as using ".1" to represent the depth value of "0.1 meters" with an abbreviated prefix, or having a variable-length header structure with an indefinite number of header lines. When existing parsing solutions encounter such non-standard formats, they directly report errors or produce incorrect parsing results, lacking adaptive correction capabilities, resulting in an overall data parsing success rate of only 60%-70%.

[0008] (iv) Lack of batch fault tolerance capability: When the existing solution handles batch file import, if a single file fails to be parsed due to abnormal format, the entire import process is often interrupted, lacking anomaly isolation and single file fault tolerance skipping mechanism. Summary of the Invention

[0009] This invention addresses the technical problems of existing static cone penetration test (CPT) data import processes, such as the need for manual specification of file formats, poor compatibility with multi-source heterogeneous formats, and lack of adaptive processing capabilities for abnormal formats. It provides a heuristic rule-based method and system for CPT file format identification and parsing. This method can automatically identify and parse CPT files with metadata in a zero-configuration manner, and import low-configuration CPT files without metadata files (only sampling interval parameters need to be provided).

[0010] To achieve the above technical objectives, this invention provides a heuristic rule-based method for CPT file format recognition and parsing, specifically including the following steps:

[0011] S1. Read the original CPT data file uploaded by the user interaction terminal line by line and store it as a list of strings to form a line sequence in memory for subsequent format detection and processing;

[0012] S2. A priority-based detection strategy is adopted to sequentially identify the format features of the line sequence in step S1 at three different granularities. The detection stops and the format type identifier is output when a match is found. The first level of detection is the full-text mark character pattern detection, and the detection result is determined to be the delimiter mark format. The second level of detection is the DAT blank line position pattern detection, and the detection result is determined to be the single-column DAT format. The third level of detection is the first-line lexical feature detection, and the detection result is determined to be the standard column format with header and the standard format without header.

[0013] S3. Based on the format type identifier output in step S2, perform the corresponding header information extraction, locate the header position, and output the hole number and data start row number;

[0014] S4. Starting from the first row number of the data, extract the pure data row sequence, select the corresponding numerical extraction method according to the format type, and perform parsing and data extraction respectively, outputting a numerical data list; the data list includes a cone resistance value list, a sidewall friction force value list, a depth value list, and a specific penetration resistance data list.

[0015] S5. Encapsulate the hole numbers output from step S3 and the data list output from step S4 into a standard data exchange format JSON for storage.

[0016] A further technical solution of the present invention: If none of the three-level detections match any format type in step S2, the file is determined to be in an unrecognizable format; and a format recognition failure identifier and file feature summary information are returned to the user's interactive terminal, prompting the user to manually specify the format type or submit the file sample to the technical support team for subsequent rule base expansion.

[0017] A further technical solution of the present invention: the full-text marker character pattern detection in step S2 is as follows: a global scan of the line sequence is performed to detect whether there are independent single-character markers "Q", "F" and "E" occupying the entire line content. When all three marker characters appear in the file, it is determined to be a delimiter marker format.

[0018] A further technical solution of the present invention: DAT blank line position pattern detection in step S2 is as follows: when the first level detection fails, DAT blank line format detection is performed; the detection condition is that three necessary conditions are met simultaneously: (a) the total number of lines in the file is not less than 13 lines; (b) there is at least one blank line in the first 15 lines; (c) the three or more consecutive lines after the first blank line are all strings that can be verified by numerical values; when all three conditions are met, it is determined to be a single-column DAT format.

[0019] A further technical solution of the present invention: In step S2, when neither the first nor the second level detection is successful, the first line lexical feature analysis is performed, and the feature judgment is made on the word sequence after the first line is segmented by whitespace characters.

[0020] If the first word is an alphanumeric combination, including hyphens and underscores, and the second word is a numeric string, including integers or numbers with decimal points, then it is determined to be a standard column format with a header; otherwise, it is determined to be a standard format without a header. For files determined to be in a standard column format with a header, the second line is further read, split, and the number of columns is determined:

[0021] When the number of columns is 3, it is determined to be a three-column format, corresponding to three columns of data: depth value, cone resistance, and sidewall friction.

[0022] When the number of columns is 2, further check whether the first word in the second row starts with ".". If it does, it is determined to be a two-column format and the depth value has an ellipsis prefix; otherwise, it is determined to be a normal two-column format.

[0023] A further technical solution of the present invention: In step S4, the data intervals are divided according to the Q / F / E marker row positions for the separator marking format to obtain the cone resistance data list and the sidewall friction data list respectively; for the single-column DAT format, the single column is read directly, and only one value is read directly for each row to obtain the specific penetration resistance data column; for the standard column format with a head and the standard format without a head, each row is divided into multiple columns according to the whitespace character to obtain the depth value list, the cone resistance list, and the sidewall friction list respectively.

[0024] The preferred technical solution of this invention is as follows: In step S4, the data extracted from the interval of the separator mark format is parsed and aligned. Specifically, the row indexes of the three marked rows Q, F, and E are accurately located, and the row interval between rows Q and F is taken as the cone resistance data list, and the row interval between rows F and E is taken as the sidewall friction data list. When the lengths of the two data lists are inconsistent, the smaller length value is taken for data truncation and alignment to ensure the consistency of the dimension in the subsequent calculation of the friction ratio, i.e., the ratio of sidewall friction to cone resistance.

[0025] The preferred technical solution of the present invention is as follows: In step S4, for the two-column format and the depth value with an omitted prefix, before data extraction, the non-standard depth value is automatically completed by prefix auto-completion. Specifically, prefix detection is performed on each word in the column where the depth value is located. If a string starting with "." is detected, "0" characters are automatically added before the string to convert it into a standard decimal string before numerical extraction.

[0026] A preferred technical solution of the present invention: The method further includes an exception handling mechanism, specifically including:

[0027] Data-level exceptions: When a data row contains an invalid string that cannot be parsed into a number, an exception containing the specific row number is thrown and the transaction operation of the current file is rolled back to prevent erroneous data from being written to the persistent storage unit.

[0028] Batch operation fault tolerance: When processing multiple files in a compressed data packet, parsing errors of a single file are captured and isolated, without affecting the normal processing of the remaining files, thus avoiding the interruption of the entire batch import process due to format errors of a single file.

[0029] The present invention also provides a system for performing a heuristic rule-based CPT file format recognition and parsing method, the system comprising a file reading module, a format detection module, a header offset parsing module, and a data extraction module;

[0030] The file reading module is used to read the original CPT data file uploaded by the user interaction terminal line by line and store it as a string list to form a line sequence in memory for subsequent format detection module processing;

[0031] The format detection module is the core processing unit. It adopts a priority-based detection strategy to sequentially identify the format features of the line sequence at three different granularities. If a match is found, the detection is terminated and a format type identifier is output.

[0032] The format detection module includes three cascaded detectors: text marker character pattern detector, DAT blank line position pattern detector, and first line lexical feature detector.

[0033] The header offset parsing module extracts the corresponding header information based on the format type identifier output by the format detection module and outputs a standardized parsing result object; the header information includes the hole number and the data start line number;

[0034] The data extraction module extracts a pure data row sequence from the row sequence based on the starting row number, performs column splitting according to the format type, extracts data from the range marked by the delimiter, and performs prefix auto-completion processing on non-standard depth values.

[0035] The data persistence storage unit writes the extracted structured data into the data persistence storage unit in the standard data exchange format (JSON), completing a fully automated processing loop from the original file to the structured data.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) This invention eliminates the manual preprocessing step and supports automatic parsing of more than 6 mainstream domestic CPT instrument output formats. For CPT files containing metadata, it achieves automatic identification and parsing with zero configuration. For header-less format files lacking metadata, users only need to provide a single parameter, the sampling interval, to complete the import. Users do not need to understand the details of the file format. The parsing time for a single file is reduced from 2-5 minutes by manual operation to the millisecond level, significantly improving the efficiency of batch data processing.

[0038] (2) This invention significantly improves format compatibility and parsing success rate. It adopts a three-level priority progressive detection, with each level using different feature vectors (character position features, blank line distribution patterns, and lexical structure features), and a highly efficient detection strategy that terminates upon hit. It achieves full spectrum coverage of format types, from high-specificity delimiter mark formats to low-specificity standard column formats, and completes format recognition in a single file scan, avoiding the performance overhead of exhaustive matching. After batch testing and verification on 500 CPT data files from different manufacturers collected in actual engineering (covering 6 mainstream formats and 3 non-standard formats such as Huakan, Qingan, and Survey Instrument Factory), the format recognition accuracy reached over 99%, and the data parsing success rate increased from the industry average of 60%-70% to over 95%.

[0039] (3) This invention possesses the capability for adaptive processing of non-standard formats; it features an automatic completion mechanism for depth values ​​from specific manufacturers that omit prefixes. For depth value strings that begin with "." to represent decimals (e.g., ".1" represents 0.1 meters), it automatically detects prefix features and adds "0" characters at the beginning, solving the problem of non-standard representation of depth values ​​encoded by instruments from specific manufacturers using an omission prefix method, without requiring customized parsing logic for individual manufacturers; by accurately locating the Q / F / E three-line marker index, it extracts the data intervals of cone resistance and sidewall friction force respectively, and automatically truncates and aligns two unequal data sequences based on the shorter one, solving the problem of inconsistent data lengths between the two channels caused by unstable instrument sampling, ensuring the dimensional consistency of friction ratio calculation. Overall, it solves the pain point of existing solutions directly reporting errors for abnormal data, enhancing the robustness and practicality of the system.

[0040] (4) The present invention has batch anomaly isolation capability and fault tolerance mechanism; through the capture and isolation mechanism of single file parsing anomaly, the abnormal file is skipped and the remaining files are processed, avoiding the interruption of the entire batch import process due to a single non-standard format file, meeting the high availability requirements of batch data processing in actual engineering, and improving the reliability of batch processing and the applicability of actual engineering. Attached Figure Description

[0041] Figure 1 This is the overall flowchart of CPT data file format detection and parsing in this invention;

[0042] Figure 2 This is a schematic diagram of the three CPT data file formats in this invention;

[0043] Figure 3 This is a flowchart of the three-level priority progressive detection process in this invention. Detailed Implementation

[0044] The present invention will be further described below with reference to specific embodiments.

[0045] This invention provides a heuristic rule-based method for CPT file format recognition and parsing, the core of which lies in a three-level priority progressive format detection algorithm. The heuristic rules described in this invention refer to format feature criteria summarized from domain knowledge, including: the occurrence pattern of marker characters, the distribution pattern of blank lines, and lexical structure features. Unlike exhaustive matching or machine learning methods, heuristic rules construct detection conditions through prior knowledge of CPT file formats from domain experts, possessing the characteristics of strong interpretability and low computational overhead. The algorithm detects in a step-by-step manner in the order of "high-specificity format → medium-specificity format → low-specificity format," using different feature vectors at each level. Upon a match, the subsequent detection terminates. The specific steps are as follows:

[0046] The system architecture and module composition in this embodiment are as follows: the system includes a file reading module, a format detection module, a header offset parsing module, and a data extraction module;

[0047] The file reading module is used to read the original CPT data file uploaded by the user interaction terminal line by line and store it as a string list to form a line sequence in memory for subsequent format detection module processing;

[0048] The format detection module is the core processing unit. It adopts a priority-based detection strategy to sequentially identify the format features of the line sequence at three different granularities. If a match is found, the detection is terminated and a format type identifier is output.

[0049] The format detection module includes three cascaded detectors: text marker character pattern detector, DAT blank line position pattern detector, and first line lexical feature detector.

[0050] The header offset parsing module extracts the corresponding header information based on the format type identifier output by the format detection module and outputs a standardized parsing result object; the header information includes the hole number and the data start line number;

[0051] The data extraction module extracts a pure data row sequence from the row sequence based on the starting row number, performs column splitting operation according to the format type, performs data extraction for the interval marked by the delimiter (Q / F / E interval), and performs prefix auto-completion processing for non-standard depth values;

[0052] The data persistence storage unit writes the extracted structured data into the data persistence storage unit in the standard data exchange format (JSON), completing a fully automated processing loop from the original file to the structured data.

[0053] The embodiment provides a heuristic rule-based method for CPT file format recognition and parsing, which specifically includes the following steps:

[0054] S1. Read the original CPT data file uploaded by the user interaction terminal line by line and store it as a list of strings to form a line sequence in memory for subsequent format detection and processing;

[0055] S2. A priority-based detection strategy is adopted to sequentially identify the format features of the row sequence in step S1 at three different granularities. If a match is found, the detection is terminated and the format type identifier is output.

[0056] The first level of detection is full-text marker character pattern detection, and the detection result is determined to be delimiter marker format (FORMAT_DELIMITER_BASED). The second level of detection is DAT blank line position pattern detection, and the detection result is determined to be single-column DAT format (FORMAT_SINGLE_COLUMN_DAT). The third level of detection is first-line lexical feature detection, and the detection result is determined to be standard column format with header and standard format without header (STANDARD_NO_HEADER).

[0057] The first-level full-text marker character pattern detection process is as follows: A global scan of the line sequence is performed to detect whether there are independent single-character markers "Q", "F", and "E" occupying the entire line content. When all three marker characters appear in the file, it is determined to be a delimiter marker format. The characteristics of this format are: the data file is divided into three independent intervals by the three marker lines Q / F / E. The area before line Q is the file header area, the area between line Q and line F is the cone resistance data area, with one value per line, and the area between line F and line E is the sidewall friction data area, with one value per line.

[0058] When the first-level detection fails, a DAT blank line format detection is performed. The detection condition requires three conditions to be met simultaneously: (a) the total number of lines in the file is not less than 13; (b) there is at least one blank line within the first 15 lines; (c) the three or more consecutive lines following the first blank line are strings that can pass numerical verification. When all three conditions are met, it is determined to be a single-column DAT format (FORMAT_SINGLE_COLUMN_DAT). The characteristics of this format are: the beginning of the file contains multiple metadata lines (including engineering information, probe specifications, calibration parameters, etc.), the data area starts from the next line after the first blank line, and the well number is fixed in the second line. Blank lines are used as a natural boundary marker between the header area and the data area. The above detection thresholds (13 lines, first 15 lines, three consecutive lines) are determined based on statistical analysis of the output formats of existing mainstream CPT instruments: the number of metadata lines in the header of mainstream DAT format files is usually in the range of 10-12 lines, the data starting line often appears within the first 15 lines, and three consecutive lines of numerical verification can effectively filter header text interference. The above thresholds can be adjusted according to the actual application scenario.

[0059] When neither the first nor the second level of detection is successful, lexical feature analysis of the first line is performed, and feature judgment is made on the word sequence after the first line is segmented by whitespace characters:

[0060] If the first word is an alphanumeric combination, including hyphens and underscores, and the second word is a numeric string, including integers or numbers with decimal points, then it is determined to be a standard column format with a header; otherwise, it is determined to be a standard format without a header (STANDARD_NO_HEADER). For files determined to be in a standard column format with a header, the second line is further read, split, and the number of columns is determined:

[0061] When the number of columns is 3, it is determined to be a three-column format (FORMAT_THREE_COLUMN_WITH_HEADER), corresponding to three columns of data: depth value, cone resistance, and sidewall friction force.

[0062] When the number of columns is 2, further check whether the first word in the second row starts with ".". If it does, it is determined to be a two-column format and the depth value has an ellipsis prefix (FORMAT_TWO_COLUMN_WITH_HEADER); otherwise, it is determined to be a normal two-column format (FORMAT_TWO_COLUMN_SIMPLE_HEADER).

[0063] If none of the three-level detections match any format type, the file is determined to be in an unrecognizable format; and a format recognition failure identifier and file feature summary information are returned to the user's interactive terminal, prompting the user to manually specify the format type or submit a file sample to the technical support team for subsequent rule base expansion.

[0064] S3. Based on the format type identifier output in step S2, perform the corresponding header information extraction, locate the header position, and output the hole number and data start line number; the hole number is used to identify which borehole the file corresponds to, and the data start line number provides step S4 with the line from which to start extracting data; these two parameters are temporarily stored in memory and passed to subsequent steps S4 and S5.

[0065] S4. Starting from the first row number of the data, extract the pure data row sequence, select the corresponding numerical extraction method according to the format type, and perform parsing and data extraction respectively to output a numerical data list; the data list includes a cone resistance value list, a sidewall friction value list, a depth value list, and a specific penetration resistance data list; the cone resistance value list will be generated in all formats, the sidewall friction value list will be generated in a three-column format and a delimiter mark format, the depth value list will be generated in a format containing a depth column, and the format without a depth list will be calculated and generated after the user provides the sampling interval.

[0066] In this step, the data intervals are segmented according to the Q / F / E marker row positions based on the separator marking format, resulting in the cone resistance data list and the sidewall friction data list, respectively. The data extracted from the intervals of the separator marking format is parsed and aligned. Specifically, the row indexes of the three marker rows Q, F, and E are precisely located, and the row interval between rows Q and F is extracted as the cone resistance data list, and the row interval between rows F and E is extracted as the sidewall friction data list. When the lengths of the two data lists are inconsistent, the smaller length value is used for data truncation and alignment to ensure the consistency of the dimension in the subsequent calculation of the friction ratio, i.e., the ratio of sidewall friction to cone resistance.

[0067] For single-column DAT format, read directly from a single column, with only one value per row, to obtain the penetration resistance data column;

[0068] For both standard column formats with and without headers, each row is divided into multiple columns by whitespace characters to obtain lists of depth values, cone resistance, and sidewall friction. When parsing a two-column format where the depth values ​​have omitted prefixes, prefix detection is performed on each token in the column containing the depth values. If a string starting with "." is detected, a "0" character is automatically added before the string to convert it into a standard decimal string before numerical parsing.

[0069] S5. Store the extracted structured data in a standard data interchange format (JSON).

[0070] The method also includes an exception handling mechanism, specifically including:

[0071] Data-level exceptions: When a data row contains an invalid string that cannot be parsed into a number, an exception containing the specific row number is thrown and the transaction operation of the current file is rolled back to prevent erroneous data from being written to the persistent storage unit.

[0072] Batch operation fault tolerance: When processing multiple files in a compressed data packet, parsing errors of a single file are captured and isolated, without affecting the normal processing of the remaining files, thus avoiding the interruption of the entire batch import process due to format errors of a single file.

[0073] The mapping table between format types and parsing strategies in this embodiment is as follows:

[0074] Format type identifier Head position rules Data start row rules Column splitting rules Special treatment FORMAT_DELIMITER_BASED All lines preceding the Q-marker line are headers; the hole number is extracted from the filename or identifier line. The line following the Q mark. Single column value After extracting the Q / F / E intervals, truncate and align them using the shorter interval. FORMAT_SINGLE_COLUMN_DAT The second line contains the hole numbers, and the area before the first blank line is the metadata section. The line after the first blank line Single column value — FORMAT_THREE_COLUMN_WITH_HEADER The first row is the header (including the hole number and markings). Second line Three parameters: depth, cone resistance, and sidewall friction. — FORMAT_TWO_COLUMN_WITH_HEADER The first row is the header (including the hole number and markings). Second line Two columns: depth value, resistance value Depth value strings that start with "." are automatically padded with a "0" prefix. FORMAT_TWO_COLUMN_SIMPLE_HEADER The first row is the header (including the hole number and markings). Second line Two columns — STANDARD_NO_HEADER Headless First line Single column value The user needs to provide the sampling interval parameter to calculate the depth value.

[0075] In this embodiment, the user terminal transmits the original CPT file to the cloud computing node via an HTTP POST request. The file receiving module writes the file to a temporary storage area and then calls a format detection engine for automatic identification. After parsing, the structured data is written to the persistent data storage unit, and the parsing results (including the identified hole numbers, data row numbers, and format type) are returned to the user terminal in JSON format. No user intervention in format selection is required throughout the entire process.

[0076] The invention will be further illustrated below with specific application examples:

[0077] Application Example 1: A scenario for automated batch processing of files in multiple formats, using a geotechnical engineering investigation project as an example. This project includes one CPT data file each from three different vendors, stored in a compressed data packet and uploaded to a cloud computing node via a user terminal. The three files are:

[0078] File A (JT109022(3)-A13.txt): Delimiter mark format. The content structure is as follows: the first line is the file name / identification information, followed by several lines of numerical values, the Nth line is the single character "Q", the area from line Q to line F is the cone resistance data area (one value per line), and the area from line F to line E is the sidewall friction data area (one value per line).

[0079] File B (DJJ1-DAT.txt): DAT blank line format. The content structure is as follows: the first line is the equipment identifier, the second line is the borehole number "ZK101", the third to eleventh lines are various metadata parameters (including probe specifications, calibration coefficients, project name, etc.), the twelfth line is blank, and the thirteenth line onwards is the specific penetration resistance data (one value per line).

[0080] File C (00J-30.txt): Three columns in standard format. The first line is "00J 30" (hole number and identifier), and each subsequent line contains three blank-separated columns of data, which are, in order, depth value, cone resistance, and sidewall friction.

[0081] The specific identification and parsing process is as follows:

[0082] S1. File decompression and reading: After receiving the compressed data packet, the cloud computing node decompresses it item by item. For each file with the suffix .txt or .dat (ignoring case differences in file extension characters), the following process is performed: all content is read line by line in character stream mode and stored as a list of strings.

[0083] S2: Format detection (taking file A as an example); The format detection module performs the first level of detection on the line list of file A: Traverse all lines, and if the 4th line contains "Q", the 9th line contains "F", and the 15th line contains "E", all three marker characters are present, and the file is immediately determined to be in FORMAT_DELIMITER_BASED format, skipping the second and third level of detection.

[0084] S3. Format Detection (taking file B as an example); The format detection module performs the first level of detection on the line list of file B: traversing all lines, no Q / F / E marker characters were found. It then proceeds to the second level of detection: the total number of lines in the file is 360 > 13 lines, satisfying the line count condition; in the first 15 lines, line 12 is detected as a blank line (index 11, counting from 0); the strings in lines 13-15 after the blank line are parsed using double-precision floating-point type verification, and all three lines are successfully parsed. All three conditions are met, and the file is determined to be in FORMAT_SINGLE_COLUMN_DAT format.

[0085] S4. Format Detection (taking file C as an example); The format detection module performed first and second level checks on the line list of file C, but both failed. It then proceeded to the third level: The first line was split by whitespace characters, resulting in the word sequence ["00J", "30"], with a word count of 2 ≥ 2; the first word "00J" passed the alphanumeric combination regular expression check, and the second word "30" passed the integer check, thus indicating a header format. The second line was read and split, resulting in 3 columns, which were determined to be FORMAT_THREE_COLUMN_WITH_HEADER.

[0086] S5. Header Offset Parsing and Data Extraction; Based on the format type of each file, perform the corresponding header offset parsing and data extraction operations:

[0087] File A: Execute the delimiter marker format header parsing, locating line Q at index 3, line F at index 8, and line E at index 14. Set the starting line number to 4 (the first line after line Q). Then perform interval data extraction: extract 4 values ​​of cone resistance from indices 4 to 7 (between Q and F), and extract 5 values ​​of sidewall friction from indices 9 to 13 (between F and E). The lengths of the two are 4 and 5 respectively; take the smaller value of 4 for truncation and alignment, discarding the 5th redundant sidewall friction value.

[0088] File B: Perform DAT blank line format header parsing, extract hole number "ZK101" from line 2 (index 1); locate blank line index as 11, and set the data start line number to 12 (first line after blank line).

[0089] File C: Perform three-column parsing including header format, extract the hole number "00J" from the first row, and set the starting row number of the data to 1 (the second row onwards is pure data).

[0090] S6. Automatic padding for non-standard depth values: When processing two-column formatted files containing depth values, such as the file "001-75.txt" in FORMAT_TWO_COLUMN_WITH_HEADER format, where the first column of each data row contains depth values, the system reads the string line by line. When a string begins with a "." (e.g., ".1", ".2"), it automatically pads it with leading zeros to obtain "0.1" and "0.2", respectively. These zeros are then converted to standard numerical values ​​using a double-precision floating-point parsing function. This mechanism ensures the numerical integrity of the depth value sequence and avoids depth data loss due to omitting prefixes.

[0091] S7. Data Structuring and Persistence: The extracted lists of cone resistance values, sidewall friction values, friction ratios calculated based on their ratios, and completed depth values ​​are encapsulated into standard data exchange format (JSON) strings and written to the corresponding fields in the drill_cpt data table. Simultaneously, the drilling progress status is updated to "CPT data imported".

[0092] Application Example 2: Application Scenarios for Headerless Standard Format Processing

[0093] When the uploaded file is a pure numerical data file (e.g., one value per line, no metadata lines), the format detection engine determines in the third-level detection that the first line's tokens do not meet the alphanumeric combination condition, directly classifying it as STANDARD_NO_HEADER format, setting the starting line number to 0, meaning the entire file is considered a pure data area. Such files typically originate from some early models of CPT instruments or are data export files that have undergone manual preprocessing. Their characteristics include a lack of metadata information such as well hole numbers and equipment identifiers, retaining only the core test data. Processing such files relies on parameters pre-configured by the user during the upload stage.

[0094] The specific processing procedure is as follows:

[0095] S1: User configures sampling parameters; the user interactive terminal provides a sampling interval configuration item on the file upload interface, and the user inputs the sampling interval value (e.g., 0.05 meters, 0.10 meters, etc.) according to the instrument settings or engineering specifications.

[0096] S2: Format detection and data extraction; After the format detection engine determines that the file is in STANDARD_NO_HEADER format, it reads the numerical string line by line starting from the first line, converts it into double-precision floating-point numbers and stores it as a data list.

[0097] S3: Automatic depth calculation; cloud computing nodes automatically calculate the depth value sequence based on the sampling interval parameters and data list length provided by the user.

[0098] The calculation rule is as follows: the depth value corresponding to the i-th row of data = sampling interval × (i + 1), where i starts counting from 0. For example, if the sampling interval is 0.05 meters and the file has a total of 200 rows of data, then the calculated depth value sequence is: the first row corresponds to 0.05 meters, the second row corresponds to 0.10 meters, ..., the 200th row corresponds to 10.0 meters.

[0099] S4: Persistent Data Storage; The calculated depth value list and the extracted numerical data list are encapsulated into a structured JSON format and written to the drill_cpt data table in the persistent data storage unit. The hole number field is set to "to be supplemented," which will be completed by the user later through the data maintenance interface.

[0100] This embodiment demonstrates the adaptive capability of the present invention when processing headerless non-standard format files. It compensates for the parsing obstacles caused by missing metadata through parameterized configuration, thus expanding the applicability of the method.

[0101] Application Example 3: Fault Tolerance in Batch Decompression Scenarios

[0102] When the compressed data package uploaded by the user terminal contains files with non-TXT / DAT extensions (such as PDF drawings, JPG site photos, etc.), extension verification is performed on each entry during decompression. Only files with the .txt and .dat extensions (ignoring case differences in file extension characters) undergo CPT format detection and parsing; other files are automatically skipped. Simultaneously, parsing anomalies in individual files during decompression (such as numerical format conversion anomalies due to unexpected data formats) are captured, anomaly information is recorded, and processing of remaining files continues, preventing the entire batch import process from being interrupted by a single non-standard file.

Claims

1. A method for recognizing and parsing CPT file format based on heuristic rules, characterized in that, Specifically, the following steps are included: S1. Read the original CPT data file uploaded by the user interaction terminal line by line and store it as a list of strings to form a line sequence in memory for subsequent format detection and processing; S2. A priority-based detection strategy is adopted to sequentially identify the format features of the line sequence in step S1 at three different granularities. The detection stops and the format type identifier is output when a match is found. The first level of detection is the full-text mark character pattern detection, and the detection result is determined to be the delimiter mark format. The second level of detection is the DAT blank line position pattern detection, and the detection result is determined to be the single-column DAT format. The third level of detection is the first-line lexical feature detection, and the detection result is determined to be the standard column format with header and the standard format without header. S3. Based on the format type identifier output in step S2, perform the corresponding header information extraction, locate the header position, and output the hole number and data start row number; S4. Starting from the first row number of the data, extract the pure data row sequence, select the corresponding numerical extraction method according to the format type, and perform parsing and data extraction respectively, outputting a numerical data list; the data list includes a cone resistance value list, a sidewall friction force value list, a depth value list, and a specific penetration resistance data list. S5. Encapsulate the hole numbers output from step S3 and the data list output from step S4 into a standard data exchange format JSON for storage.

2. The method for CPT file format recognition and parsing based on heuristic rules according to claim 1, characterized in that: If none of the three-level detections match any format type in step S2, the file is determined to be in an unrecognizable format; and a format recognition failure identifier and file feature summary information are returned to the user's interactive terminal, prompting the user to manually specify the format type or submit a file sample to the technical support team for subsequent rule base expansion.

3. The method for CPT file format recognition and parsing based on heuristic rules according to claim 1, characterized in that, The full-text marker character pattern detection in step S2 is as follows: a global scan of the line sequence is performed to detect whether there are independent single-character markers "Q", "F", and "E" occupying the entire line content. When all three marker characters appear in the file, it is determined to be a delimiter marker format.

4. The method for identifying and parsing CPT file format based on heuristic rules according to claim 1, characterized in that, The DAT blank line position pattern detection in step S2 is as follows: When the first-level detection fails, DAT blank line format detection is performed; the detection condition requires that three necessary conditions be met simultaneously: a. The total number of lines in the file shall not be less than 13; b. There is at least one blank line in the first 15 lines; c. The three or more consecutive lines following the first blank line must all be strings that pass numerical validation; If all three conditions are met, it is determined to be a single-column DAT format.

5. The method for CPT file format recognition and parsing based on heuristic rules according to claim 1, characterized in that, In step S2, if neither the first nor second-level detections are successful, the first-line lexical feature analysis is performed, and feature judgment is made on the word sequence after the first line is segmented by whitespace characters: If the first word is an alphanumeric combination, including hyphens and underscores, and the second word is a numeric string, including integers or numbers with decimal points, then it is determined to be a standard column format with a header; otherwise, it is determined to be a standard format without a header. For files determined to be in a standard column format with a header, the second line is further read, split, and the number of columns is determined: When the number of columns is 3, it is determined to be a three-column format, corresponding to three columns of data: depth value, cone resistance, and sidewall friction. When the number of columns is 2, further check whether the first word in the second row starts with ".". If it does, it is determined to be a two-column format with an ellipsis prefix in the depth value; otherwise, it is determined to be a normal two-column format.

6. The method for CPT file format recognition and parsing based on heuristic rules according to claim 1, characterized in that: In step S4, for the delimiter-marked format, the data intervals are divided according to the Q / F / E marker row positions to obtain the cone resistance data list and the sidewall friction data list respectively; for the single-column DAT format, the single column is read directly, and only one value is read directly in each row to obtain the specific penetration resistance data column; for the standard column format with a header and the standard format without a header, each row is divided into multiple columns according to the whitespace character to obtain the depth value list, the cone resistance list, and the sidewall friction list respectively.

7. A method for identifying and parsing CPT file format based on heuristic rules according to claim 2 or 6, characterized in that: In step S4, the data extracted from the delimiter-marked intervals is parsed and aligned. Specifically, the row indexes of the three marked rows Q, F, and E are precisely located, and the row interval between rows Q and F is extracted as the cone resistance data list, and the row interval between rows F and E is extracted as the sidewall friction data list. When the lengths of the two data lists are inconsistent, the smaller length value is used for data truncation and alignment to ensure the consistency of the dimension in the subsequent calculation of the friction ratio, i.e., the ratio of sidewall friction to cone resistance.

8. The method for CPT file format recognition and parsing based on heuristic rules according to claim 5, characterized in that: In step S4, for two-column formats where the depth values ​​have omitted prefixes, prefix auto-completion is performed on non-standard depth values ​​before data extraction. Specifically, prefix detection is performed on each word in the column containing the depth values; if a string starting with "." is detected, "0" characters are automatically added before the string to convert it into a standard decimal string before numerical extraction.

9. The method for identifying and parsing CPT file format based on heuristic rules according to claim 1, characterized in that, The method also includes an exception handling mechanism, specifically including: Data-level exceptions: When a data row contains an invalid string that cannot be parsed into a number, an exception containing the specific row number is thrown and the transaction operation of the current file is rolled back to prevent erroneous data from being written to the persistent storage unit. Batch operation fault tolerance: When processing multiple files in a compressed data packet, parsing errors of a single file are captured and isolated, without affecting the normal processing of the remaining files, thus avoiding the interruption of the entire batch import process due to format errors of a single file.

10. A system for executing the heuristic rule-based CPT file format recognition and parsing method according to any one of claims 1 to 8, characterized in that, The system includes a file reading module, a format detection module, a header offset parsing module, and a data extraction module; The file reading module is used to read the original CPT data file uploaded by the user interaction terminal line by line and store it as a string list to form a line sequence in memory for subsequent format detection module processing; The format detection module is the core processing unit. It adopts a priority-based detection strategy to sequentially identify the format features of the line sequence at three different granularities. If a match is found, the detection is terminated and a format type identifier is output. The format detection module includes three cascaded detectors: text marker character pattern detector, DAT blank line position pattern detector, and first line lexical feature detector. The header offset parsing module extracts the corresponding header information based on the format type identifier output by the format detection module and outputs a standardized parsing result object; the header information includes the hole number and the data start line number; The data extraction module extracts a pure data row sequence from the row sequence based on the starting row number, performs column splitting according to the format type, extracts data from the range marked by the delimiter, and performs prefix auto-completion processing on non-standard depth values. The data persistence storage unit writes the extracted structured data into the data persistence storage unit in the standard data exchange format JSON, completing a fully automated processing loop from the original file to structured data.

Images