Multi-mode text data processing system

Abstract

The invention relates to the technical field of text data processing, in particular to a text data processing system supporting multiple modes, which is suitable for batch processing of industrial logs, experimental data, financial records and other scenes, and comprises a multi-condition joint extraction module used for executing text line screening and extraction in a keyword-driven mode; the dynamic interval locking module is used for realizing continuous data interval capture in a range locking mode; the structured storage module is used for storing the extracted data in columns according to files; according to the method and the system, through the cooperative operation of keyword matching and screening and character position interception, the accuracy of target data extraction is remarkably improved in industrial equipment log processing, the false detection problem of single-condition extraction is effectively solved, and the false detection problem of single-condition extraction is solved.

Description

Technical Field

[0001] This invention relates to the field of text data processing technology, and in particular to a multi-mode text data processing system suitable for batch processing of industrial logs, experimental data, financial records and other scenarios. Background Technology

[0002] Text processing tools refer to software systems used to edit, analyze, and transform plain text or semi-structured data. Their core functions include string matching, format conversion, and data extraction. Currently widely used text processing tools can be divided into three categories: 1. Basic editor classes (such as Notepad and Vim): provide character-level editing capabilities, but lack automation capabilities; 2. Spreadsheet software (such as Microsoft Excel): Supports tabular data processing, but is limited by structured input requirements; 3. Specialized tools (such as Apache Log4j): Designed for specific domains, lacking versatility.

[0003] Summary of common defects in existing technologies: 1. Batch processing bottleneck: Most tools do not support concurrent processing of multiple files; 2. Limited extraction criteria: It cannot simultaneously satisfy both keyword matching and character position extraction requirements; 3. Missing intelligent formatting: Numerical data is more likely to be misidentified as text. 4. Dynamically capturing blanks: Interval data extraction relies on manual labeling, resulting in low automation. 5. Disorganized output management: Most users have encountered version overwrite issues. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, this application proposes to provide a multi-mode text data processing system, which adopts multi-condition joint extraction and dynamic interval locking to realize concurrent processing of multiple files.

[0005] This application discloses a multi-modal text data processing system, including: The multi-condition joint extraction module is used to perform text line filtering and extraction in keyword-driven mode; The dynamic range locking module is used to capture continuous data ranges in range locking mode; The structured storage module is used to store the extracted data in columns according to files; and, The exception handling module is used to detect input anomalies and data validity.

[0006] Furthermore, the multi-condition joint extraction module performs the following operations: S1. Keyword Matching Filter: Determines whether a text line contains one or more user-preset keywords; S2. Character Position Extraction: Extract substrings from the filtered text lines according to preset start / end character positions; S3. Data type identification: Attempt to parse the extracted substrings into numeric data and store them according to the parsing results.

[0007] Furthermore, the dynamic interval locking module adopts a state machine model, including: Detect the starting keyword in the idle state to trigger the data capture state; Cache text line data in capture mode until a terminating keyword is detected; After submitting the captured interval data, the system is reset to idle state. Optionally, a separator identifier with a preset number of rows can be inserted when submitting data.

[0008] Furthermore, the structured storage module performs the following steps: A1. Assign independent columns to each input file; A2. Write the data items from the same file into the corresponding columns in row-by-row order; A3. Physical isolation of different file data is achieved by automatically incrementing column coordinates; A4. Output file format is CSV or database table.

[0009] Furthermore, the exception handling module includes: An empty folder detection unit is used to terminate the process when the input directory is empty; The file access protection unit is used to handle abnormal file opening; and, The data validity verification unit is used to check the validity of the return value of data type conversion.

[0010] Furthermore, in the data type identification operation: When parsing is successful, the data is stored in numerical format. When parsing fails, store the data as text or execute a preset error handling strategy.

[0011] Furthermore, the interval data captured by the state machine model includes a start marker row and an end marker row.

[0012] Furthermore, the automatic incrementing of column coordinates specifically refers to: The data from the first file is written to the Nth column; The next file data is written to column N+1, achieving continuous storage across columns.

[0013] Furthermore, the data validity verification unit performs return value detection on the string-to-numerical operation, and triggers at least one of the following operations when it fails: stores the data as a text type; records an error log; or skips the current data item.

[0014] Compared with the prior art, this application has at least the following technical effects: 1. This invention significantly improves the accuracy of target data extraction in industrial equipment log processing through the collaborative operation of keyword matching and character position extraction, effectively solving the problem of false detection in single-condition extraction.

[0015] 2. The state machine-based start and end marker line capture mechanism of this invention achieves a high range integrity rate in firewall log testing.

[0016] 3. The present invention's file-based column storage and automatic column coordinate increment technology effectively avoids subsequent processing errors caused by storing data in text format, ensuring the direct computability of numerical data.

[0017] 4. The present invention employs a triple fault tolerance strategy (text storage / logging / skipping) to immediately block empty folders and prevent conversion failures, achieving a high stable operating rate in tens of thousands of stress tests.

[0018] 5. The numerical analysis and dynamic storage strategy of this invention significantly improves the success rate of digitizing and storing industrial sensor data. Attached Figure Description

[0019] Figure 1 This application includes a flowchart of the multi-condition joint extraction process. Figure 2 This application presents a dynamic interval locking state machine state transition rule diagram; Figure 3 : A schematic diagram of the structured storage in this application. Detailed Implementation

[0020] The present application will now be described in detail with reference to the accompanying drawings.

[0021] like Figure 1-3 As shown, a multi-modal text data processing system includes: The multi-condition joint extraction module is used to perform text line filtering and extraction in keyword-driven mode. The multi-condition joint extraction module performs the following operations: S1. Keyword Matching Filter: Determines whether a text line contains one or more user-preset keywords; S2. Character Position Extraction: Extract substrings from the filtered text lines according to preset start / end character positions; S3. Data type identification: Attempt to parse the extracted substring into numeric data, and store it according to the parsing results. In the data type identification operation: if the parsing is successful, the data is stored in numeric format; if the parsing fails, the data is stored as text or a preset error handling strategy is executed. A dynamic interval locking module is used to capture continuous data intervals in range locking mode. The dynamic interval locking module adopts a state machine model and includes: detecting the start keyword in the idle state and triggering the data capture state; caching text line data in the capture state until the end keyword is detected; resetting to the idle state after submitting the captured interval data; optionally, inserting a preset number of line separators when submitting data. The interval data captured by the state machine model includes a start marker line and an end marker line.

[0022] The structured storage module is used to store the extracted data in columns according to files. The structured storage module performs the following steps: A1. Assign independent columns to each input file; A2. Write the data items from the same file into the corresponding columns in row-by-row order; A3. Physical isolation of different file data is achieved by automatically incrementing column coordinates. Specifically, the automatic incrementing of column coordinates means that the first file data is written to the Nth column and the next file data is written to the N+1th column, thus achieving continuous storage across columns. A4. Output file format is CSV or database table; and, An exception handling module is used to detect input anomalies and data validity. The exception handling module includes: an empty folder detection unit, used to terminate the process when the input directory is empty; a file access protection unit, used to handle file opening anomalies; and a data validity verification unit, used to detect the validity of the return value of data type conversion. The data validity verification unit performs return value detection on string-to-numeric operations, and triggers at least one of the following operations when it fails: store as text type; record error log; skip the current data item. Example

[0023] Industrial equipment log analysis (taking CNC machining centers as an example) In a smart manufacturing scenario, it's necessary to monitor the operational status logs of 200 CNC machines in real time. Specifically, the requirement is to extract temperature parameter values ​​from the log files generated by each machine. These parameters are hidden in text lines containing the keyword "Temp:" (e.g., "Temp:36.5C"), and the numerical data needs to be extracted from the 7th to 12th characters. This system automatically scans the log files using a multi-condition extraction module. First, it filters all text lines containing the "Temp:" identifier, then precisely extracts the temperature value substring at the specified character position. The data type recognition function automatically converts substrings such as "36.5" into floating-point format for storage, while the exception handling module skips 0.01% of abnormal data (e.g., "Temp:Null") and records it in the error log. Finally, the data from the 200 machines is stored in an Excel spreadsheet using a structured storage module, with each column corresponding to the temperature sequence of a single machine. Practical application shows that this solution reduces a task that would have taken 8 hours of manual processing to within 9 minutes, improving efficiency by over 98%.

[0024] In a test of 10,000 log samples generated by 200 CNC machines, the system used a multi-condition joint extraction module to filter text lines containing the keyword "Temp:" and accurately extracted the 7th to 12th characters (e.g., extracting "36.5" from "Temp:36.5C"). Faced with 200 pre-embedded interference data entries (e.g., "Temp:Null"), the data type recognition function converted 99.6% of the valid values ​​into floating-point numbers (7,902 / 7,902), and the exception handling module performed text storage and log recording for the abnormal data. Traditional tools, unable to coordinate keyword and position extraction, missed 142 valid data entries and incorrectly stored 60 garbled characters, achieving an accuracy of only 82%. Example

[0025] Experimental data acquisition (taking a temperature sensor as an example) For the text stream data output by laboratory temperature sensors, it is necessary to capture the complete working interval data from device startup ("START") to shutdown ("END"). This system enables a dynamic interval locking module. The state machine automatically enters the capture state when it detects the "START" start marker, continuously caching the subsequent 50 lines of data until the termination marker "END" is recognized. Two lines of separators are automatically inserted before data submission to distinguish different experimental batches, and the structured storage module assigns independent columns according to sensor ID (e.g., Sensor01 column, Sensor02 column). Actual testing showed that this solution successfully solved the problem of easily losing the first and last lines of data during manual truncation, completely preserving 100% of the interval data, and improving the efficiency of experimental report generation by 85%.

[0026] For 10,000 mixed data points (such as "36.5℃") output from temperature sensors, the structured storage module stores them in columns according to sensor ID, automatically removes unit symbols, and then converts them into numerical values ​​before storing them in Excel. Tests show that the formatting accuracy of all numerical columns is 100% (0% error), while the traditional Excel text import solution cannot recognize the separation between numerical values ​​and units, resulting in 3,200 data points being incorrectly stored as text (error rate 32%), and the charting function being completely disabled. Example

[0027] Financial transaction audit The bank's system generates millions of transaction logs daily. Auditing requires extracting the transaction amount field (fixed at characters 24-30) from specific transfer records (containing the keyword "Transfer"). This system uses a multi-condition extraction module for parallel processing: first, it locates log lines containing "Transfer," then extracts the amount substring at the specified position. When encountering unusual data (such as "$1,00a"), the exception handling module triggers a triple fault tolerance mechanism—saving the original text to Excel, recording the error log, and skipping invalid entries. In a test processing 500,000 records consecutively, the system accurately extracted 99.98% of the valid transaction data, reducing the audit cycle from 5 days using the traditional method to 2 hours.

[0028] When processing 500,000 bank transaction records, the system implemented triple fault tolerance for invalid amount data such as "$1,00a": storing the original text in column B, marking the transaction number in the error log, and skipping the current item to continue execution. The numerical parsing success rate was 99.9% (499,500 / 500,000), and the fault tolerance mechanism covered 0.1% of abnormal cases; the traditional solution resulted in 340,000 data entries containing currency symbols being truncated due to forced conversion (68% failure rate), leading to severely distorted audit results. Example

[0029] Network device monitoring (taking firewall logs as an example) Network security monitoring requires the rapid extraction of DDoS attack event ranges from firewall logs. These events are marked with "Alert: DDoS" as the start marker and "End Alert" as the end marker. This system achieves automatic capture through a dynamic range locking module: the state machine caches subsequent lines upon detecting the start marker in an idle state, and submits the complete range (including start and end lines) after capturing the end marker. If the input directory is empty, the exception handling module immediately terminates the process and displays "ERR_EMPTY_DIR" to avoid resource waste. The structured storage module allocates an independent column for each attack event, retaining the original timestamp information. In a test with 100,000 log entries, the attack event extraction completeness rate reached 100%, and the security team's analysis response speed improved by 90%.

[0030] Based on real-world testing of 100,000 Cisco firewall logs, the dynamic range locking module captures attack event data between the "Alert: DDoS" and "End Alert" marked lines using a state machine mechanism. Manual sampling of 20% of the ranges (100 events) confirmed that all captured ranges completely contained the start and end marked lines, with the number of lines matching the source logs 100%. In contrast, manual operation resulted in the loss of the first and last lines of 83 events out of 500 similar events (a missing rate of 16.6%), demonstrating that this solution achieves zero data loss in dynamic range locking. Example

[0031] System stability stress test In a stress test simulating high-concurrency anomalies across 10,000 cycles (including 20 instances of empty folders, 30 instances of files with abnormal permissions, and 9,950 instances of illegal data), the anomaly handling module successfully blocked invalid processes and logged all errors. The system only experienced a write failure once under full disk load (not a design flaw), maintaining a 99.999% stable operating rate, far superior to the average 23% crash rate of traditional tools due to unhandled conversion anomalies.

Claims

1. A multi-modal text data processing system, characterized in that, include: The multi-condition joint extraction module is used to perform text line filtering and extraction in keyword-driven mode; The dynamic range locking module is used to capture continuous data ranges in range locking mode; The structured storage module is used to store the extracted data in columns according to files; and, The exception handling module is used to detect input anomalies and data validity.

2. The multi-modal text data processing system according to claim 1, characterized in that, The multi-condition joint extraction module performs the following operations: S1. Keyword Matching Filter: Determines whether a text line contains one or more user-preset keywords; S2. Character Position Extraction: Extract substrings from the filtered text lines according to preset start / end character positions; S3. Data type identification: Attempt to parse the extracted substrings into numeric data and store them according to the parsing results.

3. The multi-modal text data processing system according to claim 1, characterized in that, The dynamic interval locking module adopts a state machine model, including: Detect the starting keyword in the idle state to trigger the data capture state; Cache text line data in capture mode until a terminating keyword is detected; After submitting the captured interval data, the system is reset to idle state. Optionally, a separator identifier with a preset number of rows can be inserted when submitting data.

4. The multi-modal text data processing system according to claim 1, characterized in that, The structured storage module performs the following steps: A1. Assign independent columns to each input file; A2. Write the data items from the same file into the corresponding columns in row-by-row order; A3. Physical isolation of different file data is achieved by automatically incrementing column coordinates; A4. Output file format is CSV or database table.

5. The multi-modal text data processing system according to claim 1, characterized in that, The exception handling module includes: An empty folder detection unit is used to terminate the process when the input directory is empty; The file access protection unit is used to handle abnormal file opening; and, The data validity verification unit is used to check the validity of the return value of data type conversion.

6. A multi-modal text data processing system according to claim 2, characterized in that, In the data type identification operation: When parsing is successful, the data is stored in numerical format. When parsing fails, store the data as text or execute a preset error handling strategy.

7. A multi-modal text data processing system according to claim 3, characterized in that, The interval data captured by the state machine model includes a start marker row and an end marker row.

8. A multi-modal text data processing system according to claim 4, characterized in that, The automatic incrementing of column coordinates specifically refers to: The data from the first file is written to the Nth column; The next file data is written to column N+1, achieving continuous storage across columns.

9. A multi-modal text data processing system according to claim 5, characterized in that, The data validity verification unit checks the return value of the string-to-numerical operation. If the operation fails, it triggers at least one of the following operations: stores the data as a text type; records an error log; or skips the current data item.

Images