File analysis method and system and computing equipment

Abstract

The invention relates to the technical field of computer software, in particular to a file analysis method and system and computing device.The method comprises the steps that a universal analysis interface is defined, and a corresponding implementation class is configured for a preset file format; identifying and loading a corresponding parser class according to the file extension name; and determining the logic of data extraction and analysis through a pre-configured rule set. Aiming at the compatibility problem of a traditional tool during heterogeneous file processing, the invention provides a unified standardized analysis scheme, so that the barrier of format difference is effectively overcome, and the compatibility and flexibility of a system are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to a file parsing method, system and computing device. Background Technology

[0002] In the context of rapid information technology development, data sources are becoming increasingly diversified, and enterprises are no longer limited to processing data in a single format. However, traditional file parsing tools face significant limitations. These tools typically only support one or a few file formats, such as CSV or Excel, which is far from meeting the needs of real-world business operations for unified processing of various heterogeneous data sources (such as text files, database records, and PDF documents). Due to the lack of effective multi-format compatibility, enterprises often need to use different parsing tools when dealing with data in different formats. This not only increases the complexity of technology integration but also exacerbates the phenomenon of data silos.

[0003] Therefore, a file parsing system is needed to solve the above problems and process files in multiple formats. Summary of the Invention

[0004] Therefore, it is necessary to provide a file parsing method to address the insufficient multi-format compatibility of traditional file parsing tools, including:

[0005] Define a generic parsing interface and configure corresponding implementation classes for the preset file formats;

[0006] Identify and load the corresponding parser class based on the file extension;

[0007] The logic for data extraction and parsing is determined by a pre-configured set of rules.

[0008] Preferably, the step of identifying and loading the corresponding parser class based on the file extension includes:

[0009] The factory pattern identifies and loads the appropriate parser instance based on the file extension.

[0010] Preferably, it also includes: monitoring file changes in the target folder to determine the actual changes to the files;

[0011] Filter duplicate events when files in the target folder are modified.

[0012] The filtered data is then parsed.

[0013] Preferably, it also includes: using a double-buffered queue mechanism to separate parsing tasks from storage tasks;

[0014] The storage task is performed using batch write technology.

[0015] Preferably, the step of filtering duplicate events when determining that files in the target folder have been modified includes:

[0016] The file fingerprint algorithm filters out duplicate events when files in the target folder are modified.

[0017] Preferably, it further includes: constructing a chained data processing pipeline, and determining the data processing flow through the pre-configured rule set.

[0018] Preferably, the construction of the chain-like data processing pipeline includes:

[0019] Define a basic data processing unit and define a unified interface for the basic data processing unit;

[0020] Based on the pre-configured structured configuration file, the basic data processing is combined to form a processing chain.

[0021] Preferably, the application of batch write technology to store the data after performing the parsing task further includes:

[0022] Define a unified database write interface specification;

[0023] Calculate and adjust the amount of data to be submitted and written based on the current system load.

[0024] A second aspect of this application provides a file parsing system, comprising:

[0025] The generic parsing interface module is used to define a generic parsing interface and configure the corresponding implementation class for the preset file format.

[0026] The parser loading module is used to identify and load the corresponding parser class based on the file extension;

[0027] The rule set processing module is used to determine the logic for data extraction and parsing based on a pre-configured rule set.

[0028] A third aspect of this application provides a computing device, including: a processor; and a memory connected to the processor and storing program instructions, which, when executed by the processor, cause the processor to perform the file parsing method described above. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating a file parsing method provided in an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of the data processing flow in a file parsing method provided in an embodiment of this application;

[0031] Figure 3 This is a structural schematic diagram of the file parsing system 200 provided in an embodiment of this application;

[0032] Figure 4 This is a structural schematic diagram of a computing device 900 provided in an embodiment of this application. Detailed Implementation

[0033] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0034] It should be noted that when an element is referred to as being "set on" another element, it can be directly on the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," "top," "bottom," "end," "top," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0036] Please refer to Figures 1-2 , Figure 1 This is a flowchart illustrating a file parsing method provided in an embodiment of this application. Figure 2 This is a schematic diagram of the data processing flow in the file parsing method provided in this application embodiment. The file parsing method includes the following steps:

[0037] S101: Define a general parsing interface and configure the corresponding implementation class for the preset file format.

[0038] A standardized, universal parsing interface is constructed. This interface serves as a standardized set of functions, primarily aiming to define and standardize the fundamental functionalities necessary for all file parsers. This definition ensures consistency and interoperability in the functional implementations of different file parsers. Subsequently, for each preset file format, such as text files (TXT), comma-separated value files (CSV), portable document formats (PDF), and spreadsheets (Excel), corresponding parser implementation classes are configured. These implementation classes will specifically undertake the data parsing tasks for their respective file formats. They encapsulate dedicated logic for reading, processing, and converting data in that format, ensuring efficient and accurate extraction of the required information from various files. This modular design allows for flexible expansion to support more file formats while maintaining the overall stability and maintainability of the system.

[0039] S102: Identify and load the corresponding parser class based on the file extension.

[0040] In the runtime environment, the corresponding parser instance is dynamically loaded based on the file extension to be parsed. Furthermore, to automatically match and load the appropriate parser instance based on the file extension, this process employs a factory pattern or dependency injection mechanism, ensuring the flexibility and scalability of parser invocation. The factory pattern is a design pattern in object-oriented programming that provides an interface for creating objects, but the subclasses determine which class to instantiate. Upon receiving a file to be parsed, its extension is first extracted, and then the registry is checked to see if a matching parser class exists. If it does, the parser instance is dynamically obtained.

[0041] If a matching parser class exists, dynamically obtaining an instance of that parser can be achieved through reflection. Reflection allows the choice of which parser to use at runtime based on the file type, without needing to know all possible parser types in advance. This reduces code coupling and makes the system easier to maintain and extend.

[0042] Furthermore, the frequent creation and destruction of costly objects (such as parsers and writers) can lead to a surge in garbage collection (GC) pressure, thereby affecting system performance and stability. This implementation also introduces object pooling technology. By reusing already created objects, they can be reused across tasks, significantly reducing the frequency and lifecycle of object creation, thus lowering GC pressure. Additionally, by setting the maximum capacity of the object pool, the upper limit of memory usage can be controlled to prevent memory explosions.

[0043] S103: Determine the logic for data extraction and parsing through a pre-configured set of rules.

[0044] After determining which specific parser instance to use, but before the actual parsing is invoked, a pre-configured rule set can be introduced. This rule set can define parsing rules such as field mapping, data cleaning, and transformation logic for each file type. For example, the rule set for a .csv file can specify which columns to retain, how to handle null values, and whether to enable timestamp format conversion. This rule set can be stored in an external file or embedded in the application configuration for dynamic loading and updating.

[0045] After determining the parser instance, the specific parsing method is called based on the pre-configured rule set. Data extraction and transformation operations are performed according to the definition in the rule set. Different parsing logic can be executed according to different business needs.

[0046] Furthermore, frequent creation and destruction of temporary objects can lead to high memory consumption, heavy GC pressure, and performance degradation. This is especially true during parsing, which may generate a large number of Token objects (the smallest unit obtained after analysis, such as the token in JSONParser). Creating new objects for each instance would result in significant memory consumption. In this embodiment, a Token buffer pool is implemented using Flyweight. A new object is created only when a specific type and combination of elements appear for the first time; otherwise, existing objects are reused. This significantly reduces the creation of duplicate objects, lowers heap memory consumption, reduces GC frequency and pause time, and improves system responsiveness.

[0047] In this embodiment, by separating the rule set from the parsing logic, the parsing behavior can be easily adjusted without modifying the parser's code, enhancing the system's flexibility and maintainability. Alternatively, using an external configuration file or database to manage the rule set can be considered, allowing for dynamic updates of the parsing rules without restarting the application.

[0048] Furthermore, in this embodiment, file changes in the target folder can be monitored, duplicate events can be filtered when files in the target folder are changed, and then the filtered data can be parsed.

[0049] Specifically, to monitor file changes in the target folder, the FileSystemWatcher component can be used to continuously monitor the specified directory and capture file creation or modification operations based on the operating system-level file event notification mechanism.

[0050] The system can also determine whether a file has undergone substantial content changes by calculating a unique identifier (i.e., a "fingerprint") for its file content. When the system receives a file change event (such as "file modified"), it uses a selected hash algorithm to calculate a unique hash value as the file's "fingerprint," for example, SHA-256 can be used to calculate the hash value of the file content. This is then compared with historical fingerprints. If the newly calculated hash value does not match the old value, it is determined that there has been a genuine content change; if they match, it is considered an invalid change. Furthermore, an event deduplication algorithm is used to filter duplicate events, ensuring that the corresponding parsing task is triggered only when the file content has actually changed.

[0051] Furthermore, a cache table (which can be a memory cache or lightweight persistent storage) can be configured to record the historical hash value of each file. This embodiment can avoid redundant parsing and other repetitive processing caused by repeatedly writing the same content.

[0052] Furthermore, a chained data processing pipeline can be constructed, using the pre-configured rule set to determine the data processing flow. The chained data processing pipeline is a design pattern for building flexible and scalable data processing workflows. In this pattern, data processing tasks are broken down into a series of independent but interconnected processing units (or steps), each performing a specific data transformation or operation. These processing units are linked together in a specific order to form a "chain," where data is processed step-by-step through each processing unit in the chain.

[0053] Specifically, constructing a chained data processing pipeline includes: defining basic data processing units and defining a unified interface for each basic data processing unit; and combining the basic data processing operations into a processing chain according to a pre-configured structured configuration file. The aforementioned chained data processing unit interface includes a method definition that takes an arbitrary object as input and returns a processed result object. This allows defining various types of data processing operations as independent units, such as basic operations like string splitting, regular expression replacement, and substring extraction, which constitute the basic building blocks of more complex processing flows. The configuration file can specify the combination methods and execution order between different processing units; for example, the first level defines the splitting operation, the second level configures the regular expression replacement rules, and the third level sets the string trimming operation, etc.

[0054] For example, suppose we extract order information from a batch of CSV files and clean and format certain fields. A chained data processing pipeline can be constructed as follows:

[0055] Read a CSV file -> Use the CsvParser parser to read the raw data.

[0056] String splitting -> Separates a row of data into multiple fields using commas.

[0057] Regular expression replacement -> Apply regular expressions to certain fields to clean them up (such as removing unnecessary characters).

[0058] Numerical standardization -> Ensures that the amount field conforms to specific format requirements.

[0059] Data validation -> Checks if all required fields are complete and correctly formatted.

[0060] Write to the database -> Finally, store the cleaned data in the database.

[0061] An exception interruption mechanism and retry handling logic can also be configured. When a field fails to be processed, a preset number of retries will be performed and the result will be recorded in the error queue.

[0062] In some implementations, a double-buffered queue mechanism can be used to separate parsing and storage tasks; batch write technology is applied to execute the storage tasks. By using a double-buffered queue, parsing and storage tasks can be effectively decoupled. This means that while one thread or process is processing data parsing, another thread or process can simultaneously perform data storage operations. Specifically, when the parser parses the file content into structured data and passes it to the storage module through the double-buffered queue, the system dynamically calculates the optimal batch submission size based on the current load (e.g., CPU utilization, memory usage, etc.). The amount of data retrieved from the double-buffered queue each time is not fixed but adjusted based on real-time monitored system performance parameters.

[0063] Then, batch write technology aims to reduce the number of database interactions per transaction by submitting multiple records at once, thereby reducing database load and improving write speed. Compared to methods such as inserting records one by one, batch writes can significantly reduce the number of communications between the client and the database, thus reducing network latency.

[0064] In this implementation, the double-buffered queue mechanism first ensures that the parsed data can be efficiently transferred to the storage module, effectively avoiding performance bottlenecks that may be caused by direct synchronous calls. Subsequently, batch write technology is responsible for writing the data to the database in the most optimized way. The double-buffered queue mechanism guarantees the parallelism of the parsing and storage processes, enabling the system to maximize throughput without compromising data consistency. Batch write technology further improves the efficiency of data writing to the database and reduces the frequency of I / O (input / output) operations, thereby improving the overall system performance.

[0065] The technical solutions provided in the above implementation cases not only enable fast and efficient parsing of various file formats, but also possess effective monitoring capabilities for file changes, filtering out redundant and unnecessary repetitive parsing operations. Furthermore, the introduction of chained data processing pipelines and batch write technology further enhances data processing efficiency and overall system performance.

[0066] The embodiments described above in this application achieve accurate identification and efficient processing of file changes by defining a general parsing interface, dynamically loading parsers, combining rule sets to drive parsing logic, and introducing file fingerprinting and event deduplication mechanisms. Its beneficial effects include: improving system scalability and maintainability, reducing the development cost of new file formats; avoiding redundant parsing triggered by duplicate events with unchanged content, saving system resources; enhancing data consistency and processing efficiency, and being suitable for high-concurrency scenarios such as automated file processing, log collection, and report import.

[0067] Based on the same inventive concept, this application also provides a document parsing system 200. Please refer to... Figure 3 , Figure 3 This is a structural schematic diagram of the file parsing system 200 provided in an embodiment of this application. The file parsing system 200 includes a general parsing interface module 3210, a parser loading module 220, and a rule set processing module 330.

[0068] The generic parsing interface module 210 defines a generic parsing interface and configures corresponding implementation classes for preset file formats. Specifically, module 210 is responsible for defining a standardized interface specification, enabling different types of file parsers to be developed and integrated following a unified interface. This not only enhances the system's modularity but also improves code maintainability and extensibility. When support for new file formats needs to be added, only a new implementation class needs to be developed and ensured to implement the generic interface; no modification to existing code logic is required.

[0069] The parser loading module 220 is used to identify and load the corresponding parser class based on the file extension. In this way, different file formats can be automatically adapted without hard-coding the logic for selecting each possible parser.

[0070] The rule set processing module 230 is used to determine the logic for data extraction and parsing through a pre-configured rule set. It is responsible for managing and applying the pre-configured rule set, which guides how to extract information from the raw data and how to further process it (such as transformation and cleaning), allowing the parsing process to be customized according to business needs.

[0071] The above-described file parsing system 200 is used to execute steps S101-S103 and any of its optional embodiments. The file parsing system 200 provided in this embodiment and the embodiments of the file parsing method provided in steps S101-S103 can be referenced by each other, and will not be described again in this embodiment.

[0072] Figure 4 This is a structural schematic diagram of a computing device 900 provided in an embodiment of this application. This computing device can serve as a file parsing system, executing various optional embodiments of the above-described file parsing methods. The computing device can be a terminal, or a chip or chip system within the terminal. Figure 4 As shown, the computing device 900 includes: a processor 910, a memory 920, and a communication interface 930.

[0073] It should be understood that Figure 4 The communication interface 930 in the computing device 900 shown can be used to communicate with other devices, and may specifically include one or more transceiver circuits or interface circuits.

[0074] The processor 910 can be connected to the memory 920. The memory 920 can be used to store the program code and data. Therefore, the memory 920 can be a storage unit inside the processor 910, an external storage unit independent of the processor 910, or a component that includes both the storage unit inside the processor 910 and the external storage unit independent of the processor 910.

[0075] Optionally, the computing device 900 may also include a bus. The memory 920 and communication interface 930 can be connected to the processor 910 via the bus. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The symbol is represented by a line without an arrow, but this does not mean that there is only one bus or one type of bus.

[0076] It should be understood that in the embodiments of this application, the processor 910 may be a Central Processing Unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Alternatively, the processor 910 may employ one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0077] The memory 920 may include read-only memory and random access memory, and provides instructions and data to the processor 910. A portion of the processor 910 may also include non-volatile random access memory. For example, the processor 910 may also store device type information.

[0078] When the computing device 900 is running, the processor 910 executes computer execution instructions stored in the memory 920 to perform any of the operational steps of the above method and any of the optional embodiments thereof.

[0079] It should be understood that the computing device 900 according to the embodiments of this application can correspond to the corresponding subject in executing the methods according to the various embodiments of this application, and the above and other operations and / or functions of each module in the computing device 900 are respectively for implementing the corresponding processes of the methods of this embodiment. For the sake of brevity, they will not be described in detail here.

[0080] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0081] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device (system) embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A file parsing method, characterized in that, include: Define a generic parsing interface and configure corresponding implementation classes for the preset file formats; Identify and load the corresponding parser class based on the file extension; The logic for data extraction and parsing is determined by a pre-configured set of rules.

2. The file parsing method according to claim 1, characterized in that, The method of identifying and loading the corresponding parser class based on the file extension includes: The factory pattern identifies and loads the appropriate parser instance based on the file extension.

3. The file parsing method according to claim 2, characterized in that, Also includes: Monitor file changes in the target folder to determine the actual changes to the files; Filter duplicate events when files in the target folder are modified. The filtered data is then parsed.

4. The file parsing method according to claim 2, characterized in that, Also includes: Use a double-buffered queue mechanism to separate parsing tasks from storage tasks; The storage task is performed using batch write technology.

5. The file parsing method according to claim 2, characterized in that, The step of filtering duplicate events when determining that files in the target folder have been modified includes: The file fingerprint algorithm filters out duplicate events when files in the target folder are modified.

6. The file parsing method according to claim 1, characterized in that, Also includes: Construct a chain-like data processing pipeline and determine the data processing flow through the pre-configured rule set.

7. The file parsing method according to claim 6, characterized in that, The aforementioned construction of a chain-like data processing pipeline includes: Define a basic data processing unit and define a unified interface for the basic data processing unit; Based on the pre-configured structured configuration file, the basic data processing is combined to form a processing chain.

8. The file parsing method according to claim 4, characterized in that, The application of batch write technology to store the data after performing the parsing task also includes: Define a unified database write interface specification; Calculate and adjust the amount of data to be submitted and written based on the current system load.

9. A file parsing system, characterized in that, include: The generic parsing interface module is used to define a generic parsing interface and configure the corresponding implementation class for the preset file format. The parser loading module is used to identify and load the corresponding parser class based on the file extension; The rule set processing module is used to determine the logic for data extraction and parsing based on a pre-configured rule set.

10. A computing device, characterized in that, include: processor, and A memory connected to the processor and storing program instructions that, when executed by the processor, cause the processor to perform the file parsing method according to any one of claims 1-8.

Images