Self-adaptive file monitoring and data processing method and system based on multi-process architecture

Abstract

The invention discloses a self-adaptive file monitoring and data processing method and system based on a multi-process architecture. According to the method, a plurality of working processes are created through a process manager to monitor a specified directory in parallel; an environment detection algorithm is adopted to dynamically select a real-time monitoring or polling mode, and strategy switching and abnormal degradation during operation are supported; realizing incremental file identification and duplicate removal processing based on an MD5 file fingerprint technology; file analysis, data cleaning, quality verification and batch storage are completed through a modular data cleaning assembly line; and the process manager continuously monitors the health state of each working process and automatically restarts the abnormal process. Uniform configuration management is realized, and multi-environment deployment and hot update are supported. According to the method, the problems of single-process performance bottleneck, poor monitoring strategy adaptability, imperfect process management, insufficient increment processing capability and the like are effectively solved, multi-core CPU resources are fully utilized, and the efficiency, reliability and environmental adaptability of large-scale file monitoring and processing are remarkably improved.

Description

Technical Field

[0001] This application belongs to the field of distributed computing and data processing technology, and specifically relates to an adaptive file monitoring and data processing method, system, device, computer-readable storage medium, and electronic device based on a multi-process architecture. The solution presented in this application is mainly applied to real-time monitoring, intelligent acquisition, and automated processing scenarios of large-scale heterogeneous data sources, and is particularly suitable for file processing systems requiring high concurrency and high reliability. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, enterprises have a significantly increased demand for real-time collection and processing of massive amounts of data. In particular, in key business scenarios such as market intelligence monitoring, financial data analysis, and log processing, there is an urgent need to achieve real-time monitoring and efficient processing of large numbers of files.

[0003] Current mainstream technical solutions have the following core limitations: 1. Low efficiency of single-process processing: The traditional single-process monitoring mode cannot effectively utilize the computing power of multi-core CPUs, resulting in an inherent bottleneck in processing performance.

[0004] 2. Insufficient adaptability of monitoring strategies: Existing solutions lack the ability to dynamically select the optimal monitoring strategy based on the operating environment.

[0005] 3. Weak process management mechanism: There is a general lack of unified process lifecycle management and a sound fault recovery mechanism.

[0006] 4. Insufficient incremental processing capability: The file processing status tracking mechanism is imperfect and cannot support the incremental processing needs of large-scale files.

[0007] 5. Increased complexity in configuration management: The dispersed system parameter configurations increase the complexity of deployment and maintenance.

[0008] In addition, although real-time monitoring solutions based on watchdog libraries and periodic scanning solutions based on polling have been developed, both of these solutions have significant drawbacks: The watchdog solution heavily relies on specific library support; if the library is unavailable, the system will completely fail. Polling schemes have poor real-time performance and high resource consumption; Multi-process management mechanisms generally lack a unified creation, monitoring, and recycling process; The file processing status tracking mechanism is insufficient to handle large-scale incremental processing scenarios.

[0009] In summary, existing technical solutions are insufficient to meet the practical application needs of business scenarios with extremely high real-time requirements, such as public opinion monitoring (e.g., minute-level data collection and processing). Summary of the Invention

[0010] To address the aforementioned issues, this application proposes a novel adaptive file monitoring and data processing method based on a multi-process architecture.

[0011] This invention aims to overcome the bottlenecks of existing technologies and provide an adaptive file monitoring and data processing method and system based on a multi-process architecture. This solution achieves efficient and reliable large-scale file monitoring and processing through the synergistic effects of multi-level process management, adaptive monitoring strategy selection, incremental file processing mechanism, data cleaning pipeline, and unified configuration management.

[0012] The specific technical objectives of this invention are as follows: 1. Establish an efficient multi-process file monitoring mechanism to fully utilize multi-core CPU computing resources; 2. Implement adaptive monitoring strategy selection function, automatically matching the optimal monitoring method based on the operating environment; 3. Establish a robust process lifecycle management mechanism to ensure system stability; 4. Design an incremental file processing mechanism to avoid repetitive operations and improve overall processing efficiency; 5. Establish a unified configuration management framework to reduce the complexity of system deployment and maintenance.

[0013] To achieve the above objectives, the present invention employs the following technical strategy: (1) Multi-process adaptive monitoring mechanism It pioneered an adaptive monitoring strategy selection algorithm based on environmental detection, which automatically selects the optimal monitoring mode according to the system environment, solving the compatibility problem of traditional file monitoring systems in different environments and ensuring the stability and continuity of monitoring.

[0014] Specific implementation methods: The environment detection algorithm detects the operating system type, watchdog library availability, and file system event support; the intelligent strategy selects between real-time monitoring and polling monitoring modes; and the dynamic switching mechanism enables abnormal degradation and automatic recovery.

[0015] (2) Incremental file processing mechanism Based on the intelligent incremental processing mechanism of MD5 file fingerprint technology, it achieves efficient deduplication and breakpoint resume, significantly improving the efficiency of large-scale file processing and reducing unnecessary computational overhead.

[0016] Specific implementation: File fingerprint generation uses the formula Fingerprint=MD5(FilePath+str(FileSize)+str(LastModified)); File processing status is recorded through the FileStatusStore module, supporting resume download after system restart; Incremental update strategy only processes new files or modified files.

[0017] (3) Unified configuration management framework The system is designed with a unified configuration management mechanism to support multi-environment deployment and dynamic configuration updates, simplifying system deployment and maintenance and improving system manageability.

[0018] Specific implementation methods: centralized configuration management of all monitoring directories, processing parameters, database connections, etc.; supports configuration switching between multiple environments such as development, testing, and production; supports runtime configuration updates without restarting the system.

[0019] (4) Process health monitoring and automatic recovery Based on the process health monitoring and automatic recovery mechanism of the multiprocessing module, it provides enterprise-level high availability assurance and reduces the need for manual intervention.

[0020] The specific implementation method is as follows: ProcessManager monitors the health status of all DirectoryWorker processes in real time; detects abnormal situations such as process crashes and zombies; and automatically restarts abnormal processes to ensure high system availability.

[0021] Specifically, this application provides the following technical solutions: The first aspect of this application provides an adaptive file monitoring and data processing method based on a multi-process architecture, the method comprising: S1. Multi-process initialization and configuration management: When the system starts, a corresponding number of independent worker processes are created according to the number of CPU cores. Each worker process is responsible for monitoring a specified directory. The system configuration is loaded through a unified configuration management framework. The system configuration includes the monitoring directory, processing parameters and database connection information. S2. Adaptive monitoring strategy selection: Based on the analysis of operating environment parameters using an environment detection algorithm, the optimal monitoring mode is dynamically selected; the monitoring modes include a real-time monitoring mode based on file system events and a polling mode based on time intervals; when an abnormality occurs in the monitoring mode, a dynamic switching mechanism is triggered, automatically downgrading to the polling mode and supporting recovery; S3. Incremental file processing: An incremental file processing mechanism based on file fingerprints is adopted. By comparing the fingerprint of the new file with the historical records, newly added or modified files are identified and processed to avoid repeated operations on already processed files. S4. Data Processing Pipeline: Performs data cleaning, quality verification, format conversion, and batch data entry operations on the identified files, completing the entire process from data acquisition to storage.

[0022] Furthermore, in the method of this application, the multi-process initialization in step S1 specifically includes: Create a process manager and set the maximum number of worker processes to the number of CPU cores; Establish an inter-process communication mechanism, including a result queue for passing processing results and a stop event for coordinating process termination; Create a separate DirectoryWorker process for each monitored directory to achieve process-level isolation; The system configuration is loaded through a centralized configuration management module, supporting YAML format files, environment variables, and runtime dynamic updates.

[0023] Furthermore, in the method of this application, the adaptive monitoring strategy selection in step S2 specifically includes: Environment detection algorithm: detects operating system type, watchdog library availability, and file system event support; Strategy selection logic: If watchdog is supported and the file system supports event-driven operation, select real-time monitoring mode; otherwise, select polling mode. Dynamic switching mechanism: The monitoring mode is automatically switched according to the system status during runtime. In case of an anomaly, it is downgraded to polling mode and automatically upgraded to real-time monitoring mode after the environment is restored.

[0024] Furthermore, in the method of this application, the incremental file processing mechanism in step S3 adopts MD5-based file fingerprinting technology, and the file fingerprint generation algorithm formula is: Fingerprint = MD5(ContentHash || ModifiedTimeHash || FileSizeHash); Where ContentHash is the MD5 hash of the file content, ModifiedTimeHash is the MD5 hash of the last modified time, FileSizeHash is the MD5 hash of the file size, and || represents the string concatenation operation.

[0025] Furthermore, in the method of this application, the incremental file processing mechanism in step S3 adopts MD5-based file fingerprinting technology, and the file fingerprint generation algorithm formula is: Fingerprint = MD5(FileName || FileSize || LastModifiedTime ||First1KBContent); Where FileName is the filename, FileSize is the file size, LastModifiedTime is the last modification time, First1KBContent is the first 1024 bytes of the file, and || represents the string concatenation operation.

[0026] Furthermore, in the method of this application, the incremental file processing in step S3 further includes: The FileStatusStore module records file processing status and supports resuming interrupted downloads after system restart. An incremental update strategy is adopted, and processing operations are only performed on new files or files whose fingerprints have changed.

[0027] Furthermore, in the method of this application, the data processing pipeline in step S4 specifically includes: Data Acquisition and Analysis: Identify file formats based on file extensions and content features, parse CSV, JSON, and XML content, and detect file encoding; Data cleaning and validation: handling missing values ​​and outliers, performing data type conversion and deduplication; Data conversion and storage: Perform format standardization, field mapping, batch writing to the database, and record the processing status.

[0028] Furthermore, the method of this application also includes: Process health monitoring and automatic recovery: Real-time monitoring of the running status of all worker processes. When a process crash or zombie is detected, the abnormal process is terminated and a new process is automatically restarted based on the original configuration parameters, thus achieving self-healing.

[0029] A second aspect of this application provides an adaptive file monitoring and data processing system based on a multi-process architecture. The system, during runtime, implements the steps of the aforementioned adaptive file monitoring and data processing method based on a multi-process architecture. The system includes: Command line interface layer: used to parse command line parameters, serving as the system startup entry point; Process management layer: This includes the process manager, which is responsible for creating, monitoring, and recycling worker processes, and for implementing process health monitoring and automatic recovery; Worker process layer: contains multiple independent DirectoryWorker processes, each responsible for monitoring a directory and configuring an independent data processing pipeline; Monitoring strategy layer: includes monitoring strategy selector, watchdog event monitor and polling monitor, which dynamically selects the monitoring mode based on environmental detection results; Data processing layer: includes components for data parsing, cleaning, validation, transformation, and data storage; Communication and State Layer: Includes result queues, stop events, and file state storage, used for inter-process communication and incremental processing state tracking.

[0030] A third aspect of this application provides an adaptive file monitoring and data processing apparatus based on a multi-process architecture. The apparatus, when running, implements the steps of the aforementioned adaptive file monitoring and data processing method based on a multi-process architecture. The apparatus includes: The multi-process initialization and configuration management module is used to create a corresponding number of independent worker processes based on the number of CPU cores when the system starts up. Each worker process is responsible for monitoring a specified directory. The system configuration is loaded through a unified configuration management framework. The system configuration includes the monitoring directory, processing parameters, and database connection information. An adaptive monitoring strategy selection module is used to analyze operating environment parameters based on an environment detection algorithm and dynamically select the optimal monitoring mode. The monitoring modes include a real-time monitoring mode based on file system events and a polling mode based on time intervals. When an abnormality occurs in the monitoring mode, a dynamic switching mechanism is triggered to automatically downgrade to the polling mode and support recovery. The incremental file processing module is used to adopt an incremental file processing mechanism based on file fingerprints. By comparing the fingerprint of the new file with the historical records, it identifies newly added or modified files for processing, avoiding repeated operations on already processed files. The data processing pipeline module is used to perform data cleaning, quality verification, format conversion, and batch storage operations on the identified files, completing the entire process from data acquisition to storage.

[0031] A fourth aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned adaptive file monitoring and data processing method based on a multi-process architecture.

[0032] The fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned adaptive file monitoring and data processing method based on a multi-process architecture.

[0033] In summary, compared with existing technologies, this invention has significant advantages in terms of technical design and practical application effects: (1) Performance advantages: The multi-process architecture makes full use of multi-core CPU resources, improving file processing performance by 3-5 times; intelligent resource management dynamically adjusts the number of processes according to system load; incremental processing mechanism avoids redundant calculations and significantly improves data processing efficiency.

[0034] (2) Reliability advantages: Adaptive monitoring strategy ensures stable and reliable operation of the system under various operating environments; process health monitoring monitors the process status in real time and automatically restarts abnormal processes; data integrity guarantee ensures data quality through a complete data cleaning and verification process.

[0035] (3) Maintainability advantages: Unified configuration management simplifies the system deployment process and reduces the complexity of operation and maintenance; modular architecture design supports flexible functional expansion to meet the needs of different application scenarios; detailed log recording facilitates problem investigation and system maintenance.

[0036] Practical application effect verification: The system of this invention has been successfully deployed and applied in the business intelligence monitoring system. After actual operation verification, it has achieved significant technical effects: (1) Significantly improved processing performance A single server can monitor more than 20 directories simultaneously and process more than 1 million files per day; Compared to traditional single-process solutions, file processing throughput is increased by 3-5 times; The multi-process architecture makes full use of the server's multi-core CPU resources, increasing CPU utilization from 20% to 80%.

[0037] (2) Real-time performance has been greatly improved Data acquisition latency has been reduced from hours in traditional solutions to minutes. Adaptive monitoring strategies ensure that file changes are detected and processed in a timely manner; Incremental processing avoids redundant calculations, further shortening data processing time.

[0038] (3) Enhanced system reliability The system operated continuously and stably for 30 days without any failures, achieving a system availability of 99.9%. ProcessManager's process health monitoring mechanism ensures that abnormal processes are automatically recovered; The adaptive monitoring strategy dynamic switching function ensures stable operation of the system in different environments.

[0039] (4) Optimization of resource utilization efficiency Memory usage is reduced by 40% compared to traditional solutions, and resource utilization is significantly improved. Incremental processing mechanisms avoid redundant file processing, saving computing resources and storage space.

[0040] Unified configuration management simplifies system operation and maintenance and reduces labor costs.

[0041] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of this application, the accompanying drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the overall implementation of the adaptive file monitoring and data processing method based on a multi-process architecture, as described in this application.

[0044] Figure 2 This is the overall architecture diagram of the adaptive file monitoring and data processing system based on a multi-process architecture, as described in this application.

[0045] Figure 3 This is a flowchart of the process management algorithm used in the embodiments of this application.

[0046] Figure 4 This is a diagram of the adaptive monitoring selection algorithm used in the embodiments of this application.

[0047] Figure 5 This is a diagram of the incremental file processing algorithm used in the embodiments of this application.

[0048] Figure 6 This is a structural diagram of the adaptive file monitoring and data processing device based on a multi-process architecture, as described in this application.

[0049] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0051] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0052] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0053] Figure 1 The diagram shows the overall implementation flow of the adaptive file monitoring and data processing method based on a multi-process architecture provided in this application, including the following steps: S1. Multi-process initialization and configuration management: When the system starts, a corresponding number of independent worker processes are created according to the number of CPU cores. Each worker process is responsible for monitoring a specified directory. The system configuration is loaded through a unified configuration management framework. The system configuration includes the monitoring directory, processing parameters and database connection information. S2. Adaptive monitoring strategy selection: Based on the analysis of operating environment parameters using an environment detection algorithm, the optimal monitoring mode is dynamically selected; the monitoring modes include a real-time monitoring mode based on file system events and a polling mode based on time intervals; when an abnormality occurs in the monitoring mode, a dynamic switching mechanism is triggered, automatically downgrading to the polling mode and supporting recovery; S3. Incremental file processing: An incremental file processing mechanism based on file fingerprints is adopted. By comparing the fingerprint of the new file with the historical records, newly added or modified files are identified and processed to avoid repeated operations on already processed files. S4. Data Processing Pipeline: Performs data cleaning, quality verification, format conversion, and batch data entry operations on the identified files, completing the entire process from data acquisition to storage.

[0054] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0055] Example 1: An Adaptive File Monitoring and Data Processing Method Based on a Multi-Process Architecture Step 1: Multi-process initialization and configuration management During system startup, the ProcessManager module creates a corresponding number of independent DirectoryWorker processes based on the number of CPU cores. Each process is responsible for monitoring a specified directory. Configuration information is centrally managed through the unified app_config module, which supports YAML format configuration files.

[0056] Specific implementation steps: (1) Process Manager initialization: The ProcessManager module creates a process pool based on the multiprocessing module and sets the maximum number of working processes to the number of CPU cores.

[0057] (2) Communication mechanism establishment: Create multiprocessing.Queue for inter-process result transfer, and multiprocessing.Event for coordinating process termination.

[0058] (3) Creation of worker processes: Create an independent DirectoryWorker process for each monitored directory to achieve process-level isolation.

[0059] (4) Configuration loading: The system configuration is loaded uniformly through the app_config module, supporting environment variables and configuration files.

[0060] Algorithm implementation pseudocode: Algorithm 1: Multi-process initialization algorithm Input: CPU core count max_workers, list of monitored directories.

[0061] Output: ProcessManager, which has been initialized.

[0062] step: 1. Create a process manager instance and set the maximum number of worker processes to the number of CPU cores; 2. Establish inter-process communication mechanisms: result queues and stop events; 3. Create a DirectoryWorker process for each monitored directory; 4. Load a unified configuration file, supporting environment variables and YAML format; 5. Return to the process manager after initialization is complete.

[0063] Algorithm 2: Environmental Detection Algorithm Input: None.

[0064] Output: Boolean value indicating whether watchdog monitoring is supported.

[0065] step: 1. Try importing the watchdog library; 2. Return True if the import is successful; 3. Return False if import fails.

[0066] Figure 3 The diagram shows the process management algorithm flow used in this embodiment. The diagram illustrates the initialization and management process of the ProcessManager module, which specifically includes the following steps: (1) ProcessManager initialization: Load configuration parameters and prepare for the multi-process environment; (2) Set the maximum number of processes: Determine the maximum number of working processes based on the number of CPU cores; (3) Create communication components: Establish a result queue and stop event for inter-process communication; (4) Initialize global state: Establish file status tracking and statistical information recording; (5) Add DirectoryWorker: Create a worker process for each monitored directory; (6) Verify directory existence: Ensure the monitored directory is valid and accessible; (7) Check process limit: Verify that the current number of processes does not exceed the system limit; (8) Create a DirectoryWorker instance: Instantiate the worker process object; (9) Start worker processes: Activate all DirectoryWorker processes; (10) Enter the monitoring loop: ProcessManager enters the main monitoring loop and continuously monitors the process status.

[0067] Step Two: Selection and Implementation of Intelligent Monitoring Strategies The system automatically selects the optimal monitoring strategy based on the operating environment: event-driven monitoring is used in environments that support watchdog timers, while polling monitoring is used in environments that do not. The environment detection algorithm makes intelligent judgments based on system type, file system characteristics, and available libraries.

[0068] Specific implementation steps: (1) Environmental detection algorithm: Detect operating system type (Windows / Linux / macOS); Check if the watchdog library is available; Verify whether the file system supports inotify (Linux) or ReadDirectoryChangesW (Windows); Assess system resource limitations and performance requirements.

[0069] (2) Monitoring strategy selection logic: Event-driven pattern: When a watchdog is available and the file system supports it, use the Observer pattern to monitor file changes in real time; Polling mode: When event-driven operations are unavailable, periodic directory scans are performed based on the configured scan interval; Hybrid mode: Combines two modes for optimal performance in partially supported environments.

[0070] (3) Adaptive switching mechanism: The monitoring strategy can be dynamically adjusted based on the system status during runtime; When an event-driven monitoring anomaly occurs, it automatically degrades to polling mode. Configuration parameters are provided to allow users to manually specify policies.

[0071] Algorithm implementation pseudocode: Algorithm 3: Adaptive Monitoring Strategy Selection Algorithm Input: System configuration (config).

[0072] Output: Instance of optimal monitoring strategy.

[0073] step: 1. Detect the operating system type and file system characteristics; 2. Verify the availability of the watchdog library; 3. Check the file system's event support capabilities; 4. Select a strategy based on the test results: - If watchdog is supported and the file system supports events: Select event-driven mode; - If the user explicitly specifies polling: Select polling mode; - Other cases: Select polling mode (best compatibility); 5. Return to the selected monitoring policy instance.

[0074] Figure 4 This diagram illustrates the adaptive monitoring selection algorithm used in this embodiment. The diagram shows the dynamic switching mechanism of the adaptive monitoring strategy, and the specific state transitions include: (1) Environment detection: The system detects the operating environment when it starts up and determines whether the watchdog library is available; (2) Strategy selection: Select real-time monitoring or polling monitoring mode based on the detection results; (3) Dynamic switching: The monitoring strategy is automatically adjusted according to changes in the environment during operation; (4) Anomaly handling: When an anomaly is detected, the system automatically degrades to polling mode to ensure continuous system operation; (5) Recovery mechanism: After the environment is restored, it will automatically upgrade to real-time monitoring mode to improve monitoring efficiency.

[0075] Step 3: Incremental File Processing and Intelligent Deduplication Mechanism The system uses MD5-based file fingerprinting technology to achieve efficient incremental processing. It ensures the uniqueness and stability of fingerprints through multi-dimensional feature combinations, avoiding the repeated processing of the same files.

[0076] Specific implementation steps: (1) File fingerprint generation algorithm: Content hashing: Calculates the MD5 value of the file content to ensure that the fingerprint changes when the content changes; Timestamp feature: Records the last modification time of a file, detecting file updates; File size feature: Contains file size information, enabling quick detection of file changes; Combined fingerprint: Combining three features into a unique identifier.

[0077] (2) Incremental processing logic: Fingerprint database: Maintains fingerprint records of processed files; Change detection: Compares new fingerprints with historical records to identify newly added / modified files; Deduplication: Skip files that match fingerprints and only process new files; Status management: Tracks file processing status and supports resuming interrupted downloads.

[0078] (3) Performance optimization strategies: Fingerprint caching: Caches the fingerprints of recently processed files to reduce redundant calculations; Batch processing: Supports batch file fingerprint generation and comparison; Incremental update: Only the changed parts are recalculated.

[0079] File fingerprint generation algorithm formula: Fingerprint = MD5(ContentHash || ModifiedTimeHash || FileSizeHash) in: - ContentHash = MD5(file content); - ModifiedTimeHash = MD5 (last modified timestamp); - FileSizeHash = MD5 (file size); - || indicates a string concatenation operation.

[0080] Algorithm time complexity analysis: File reading: O(n), where n is the file size; MD5 calculation: O(1) (fixed-length hash); Fingerprint comparison: O(1) (hash table lookup); Overall complexity: O(n), mainly depends on the file size.

[0081] Algorithm implementation pseudocode: Algorithm 4: File Fingerprint Generation and Incremental Processing Algorithm Input: file path file_path.

[0082] Output: Boolean value indicating whether the file has changed.

[0083] step: 1. Generate file fingerprint: - Get file metadata (size, modification time); - Read the first 1KB of the file; - Calculate the MD5 hash value: Fingerprint = MD5(filename + file size + modification time + first 1KB content).

[0084] 2. Check for file changes: - Retrieve historical fingerprints from the cache; - If no record is found in the cache, mark it as a changed file and cache the new fingerprint; - If the fingerprints are inconsistent, mark it as a changed file and update the cache; - If the fingerprints match, mark it as an unchanged file.

[0085] 3. Returns the file change status.

[0086] File fingerprint calculation formula: F = MD5(file_name || file_size || mtime || first_1KB) in: file_name: File name; file_size: File size (bytes); mtime: Last modified time (timestamp); first_1KB: The first 1024 bytes of the file.

[0087] Figure 5 This diagram illustrates the incremental file processing algorithm used in this embodiment. The diagram shows the entire process of incremental file processing, including the following steps: (1) File discovery: The monitoring system detects new files or file changes; (2) Generate file fingerprint: Use the MD5 algorithm to generate a unique identifier for the file; (3) Query Status Storage: Check the file processing status in FileStatusStore; (4) Determine the processing status: Decide whether to process the data based on the status; (5) Data processing pipeline: Perform CSV parsing, data cleaning, and database insertion on unprocessed files; (6) Status update: Update the file status after processing to avoid duplicate processing; (7) Statistical information update: record processing results and system operation statistics.

[0088] Step 4: Data Cleaning and Inbound Processing Line The system adopts a modular data processing pipeline design, with each DirectoryWorker process configured with an independent DataPipeline instance to realize the entire process of data processing from collection to storage.

[0089] Specific implementation steps: (1) Data acquisition and analysis: File type identification: Automatically identifies formats such as CSV, JSON, and XML based on file extensions and content characteristics; Data parsing: Use pandas, csv modules, etc. to parse file content and convert it into structured data; Encoding detection: Automatically detects file encoding format, supporting multiple encodings such as UTF-8 and GBK.

[0090] (2) Data cleaning and quality verification: Missing value handling: Identifies and handles null and NaN values, supporting strategies such as filling and deletion; Data type conversion: Automatically identifies and converts data types to ensure data format consistency; Outlier detection: Identifies abnormal data based on statistical methods, supporting threshold filtering and correction; Data deduplication: Deduplicating data based on key fields to avoid duplicate data being entered into the database.

[0091] (3) Data transformation and standardization: Standardize formats: unify the formats of dates, times, and numbers; Field mapping: Rename and map fields according to configuration rules; Data aggregation: Supports data aggregation calculations based on time windows.

[0092] (4) Data import and status management: Batch write: Improve database write performance by using batch insert. Transaction management: Ensures the atomicity and consistency of data writes; Status tracking: Records the processing status and data entry time for each file.

[0093] Data processing pipeline architecture: File acquisition → Format parsing → Data cleaning → Quality verification → Format conversion → Batch import Data cleaning algorithm formula: Data quality score = (Number of valid data records / Total number of records) × Weight; Cleaning success rate = (number of successfully processed records / total number of records) × 100%.

[0094] Algorithm implementation pseudocode: Algorithm 5: Data Processing Pipeline Algorithm Input: file path (file_path), configuration parameter (config).

[0095] Output: Boolean value indicating whether the processing was successful.

[0096] step: 1. File parsing stage: - Recognize file formats (CSV, JSON, XML, etc.); - Read file contents and parse them into structured data; - Detect file encoding and delimiters.

[0097] 2. Data cleaning stage: - Remove null values ​​and invalid records; - Standardize field formats (date, number, string); - Handle missing and outlier values.

[0098] 3. Data Validation Phase: - Check data integrity and consistency; - Validate business rule constraints; - Quality indicators of test data.

[0099] 4. Data entry stage: - Connect to the target database; - Perform batch insert operations; - Handling insertion exceptions and retry mechanisms.

[0100] 5. Return the processing result status.

[0101] Data cleaning algorithm formula: Cleaned data = {Records ∈ Original data | Integrity (records) ∧ Validity (records) ∧ Consistency (records)} in: Completeness: The record contains all required fields; Validation: Field values ​​conform to predefined rules; Consistency: The relationships between fields conform to the business logic.

[0102] Example 2: An Adaptive File Monitoring and Data Processing System Based on a Multi-Process Architecture Figure 2 This is the overall architecture diagram of the system. The diagram shows the six-layer architecture design of the adaptive file monitoring and data processing system based on a multi-process architecture, as described in this application. Specifically, it includes: (1) Command line interface layer: Command line argument parsing: Responsible for parsing command line arguments entered by the user.

[0103] Main program entry point: The system startup entry point, coordinating the initialization of various modules.

[0104] (2) Process Management Layer: ProcessManager: The core management module, responsible for the initialization and management of the multi-process environment.

[0105] Worker process management: Responsible for the creation and destruction of DirectoryWorker processes.

[0106] Process health monitoring: Real-time monitoring of the health status of worker processes, enabling automatic restart of abnormal processes.

[0107] (3) Work process layer: DirectoryWorker processes 1-n: Multiple independent worker processes, each responsible for monitoring a directory.

[0108] Inter-process isolation: Each DirectoryWorker process runs independently and does not affect others.

[0109] (4) Monitoring strategy layer: Monitoring strategy selector: Selects the optimal monitoring strategy based on environmental monitoring results.

[0110] Watchdog event monitoring: Real-time file system event monitoring based on the watchdog library.

[0111] Polling monitor: A polling monitoring mode based on time intervals.

[0112] (5) Data processing layer: DataPipeline instances 1-n: Data processing pipelines, each DirectoryWorker is configured with an independent instance.

[0113] Data processing components include CSV parsing, data cleaning, quality verification, and database insertion.

[0114] (6) Communication and State Layer: Result queue: An inter-process communication component used to transmit processing results and status information.

[0115] Stop event: System stop signal, coordinating the graceful shutdown of all processes.

[0116] FileStatusStore: File status storage, enabling incremental processing based on file fingerprint technology.

[0117] Example 3: Implementation of an Adaptive File Monitoring and Data Processing Method Based on a Multi-Process Architecture (a) System startup process (1) Parameter parsing and configuration loading Parse command-line arguments to determine the monitoring directory and operating mode; Load the configuration file and set system parameters.

[0118] (2) ProcessManager initialization Set the maximum number of working processes based on the number of CPU cores; Create inter-process communication components (result queue, stop event); Initialize the global state tracking system.

[0119] (3) DirectoryWorker process creation Create a separate DirectoryWorker process for each monitored directory; Each process is configured with an independent DataPipeline instance; Start all worker processes.

[0120] (II) Operational Phase Process (1) Document monitoring and discovery Each DirectoryWorker performs file monitoring according to the configured monitoring mode; Once a file change is detected, the incremental processing algorithm is invoked.

[0121] (2) Data processing pipeline CSVParser parses file content; DataCleaner performs data cleaning and quality verification; DorisWriter writes data to the database.

[0122] (3) Status tracking and reporting FileStatusStore updates file processing status; The processing results are reported to ProcessManager via a queue; Generate real-time statistical reports.

[0123] Figure 6 The image shows an adaptive file monitoring and data processing device based on a multi-process architecture proposed in this application. The device includes: The multi-process initialization and configuration management module is used to create a corresponding number of independent worker processes based on the number of CPU cores when the system starts up. Each worker process is responsible for monitoring a specified directory. The system configuration is loaded through a unified configuration management framework. The system configuration includes the monitoring directory, processing parameters, and database connection information. An adaptive monitoring strategy selection module is used to analyze operating environment parameters based on an environment detection algorithm and dynamically select the optimal monitoring mode. The monitoring modes include a real-time monitoring mode based on file system events and a polling mode based on time intervals. When an abnormality occurs in the monitoring mode, a dynamic switching mechanism is triggered to automatically downgrade to the polling mode and support recovery. The incremental file processing module is used to adopt an incremental file processing mechanism based on file fingerprints. By comparing the fingerprint of the new file with the historical records, it identifies newly added or modified files for processing, avoiding repeated operations on already processed files. The data processing pipeline module is used to perform data cleaning, quality verification, format conversion, and batch storage operations on the identified files, completing the entire process from data acquisition to storage.

[0124] When the above-mentioned device is in operation, it implements the steps of the adaptive file monitoring and data processing method based on a multi-process architecture disclosed in this application.

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. Each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0126] like Figure 7As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned adaptive file monitoring and data processing method based on a multi-process architecture.

[0127] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0128] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0129] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0130] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0131] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the adaptive file monitoring and data processing method based on a multi-process architecture disclosed in this application.

[0132] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0133] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0134] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0135] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0136] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0137] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0138] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0139] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0140] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0141] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0142] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0143] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the adaptive file monitoring and data processing method based on a multi-process architecture disclosed in this application. When the computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.

[0144] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0145] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An adaptive file monitoring and data processing method based on a multi-process architecture, characterized in that, The method includes: S1. Multi-process initialization and configuration management: When the system starts, a corresponding number of independent worker processes are created according to the number of CPU cores. Each worker process is responsible for monitoring a specified directory. The system configuration is loaded through a unified configuration management framework. The system configuration includes the monitoring directory, processing parameters and database connection information. S2. Adaptive monitoring strategy selection: Based on the analysis of operating environment parameters using an environment detection algorithm, the optimal monitoring mode is dynamically selected; the monitoring modes include a real-time monitoring mode based on file system events and a polling mode based on time intervals; when an abnormality occurs in the monitoring mode, a dynamic switching mechanism is triggered, automatically downgrading to the polling mode and supporting recovery; S3. Incremental file processing: An incremental file processing mechanism based on file fingerprints is adopted. By comparing the fingerprint of the new file with the historical records, newly added or modified files are identified and processed to avoid repeated operations on already processed files. S4. Data Processing Pipeline: Performs data cleaning, quality verification, format conversion, and batch data entry operations on the identified files, completing the entire process from data acquisition to storage.

2. The method according to claim 1, characterized in that, The multi-process initialization in step S1 specifically includes: Create a process manager and set the maximum number of worker processes to the number of CPU cores; Establish an inter-process communication mechanism, including a result queue for transmitting processing results and a stop event for coordinating process termination; Create a separate DirectoryWorker process for each monitored directory to achieve process-level isolation; The system configuration is loaded through a centralized configuration management module, supporting YAML format files, environment variables, and runtime dynamic updates.

3. The method according to claim 1, characterized in that, The adaptive monitoring strategy selection in step S2 specifically includes: Environment detection algorithm: detects operating system type, watchdog library availability, and file system event support; Strategy selection logic: If watchdog is supported and the file system supports event-driven operation, select real-time monitoring mode; otherwise, select polling mode. Dynamic switching mechanism: The monitoring mode is automatically switched according to the system status during runtime. In case of an anomaly, it is downgraded to polling mode and automatically upgraded to real-time monitoring mode after the environment is restored.

4. The method according to claim 1, characterized in that, The incremental file processing mechanism described in step S3 uses MD5-based file fingerprinting technology. The file fingerprint generation algorithm formula is as follows: Fingerprint = MD5(ContentHash || ModifiedTimeHash || FileSizeHash); Where ContentHash is the MD5 hash of the file content, ModifiedTimeHash is the MD5 hash of the last modified time, FileSizeHash is the MD5 hash of the file size, and || represents the string concatenation operation.

5. The method according to claim 1, characterized in that, The incremental file processing mechanism described in step S3 uses MD5-based file fingerprinting technology. The file fingerprint generation algorithm formula is as follows: Fingerprint = MD5(FileName || FileSize || LastModifiedTime ||First1KBContent); Where FileName is the filename, FileSize is the file size, LastModifiedTime is the last modification time, First1KBContent is the first 1024 bytes of the file, and || represents the string concatenation operation.

6. The method according to claim 1, characterized in that, The incremental file processing in step S3 further includes: The FileStatusStore module records file processing status and supports resuming interrupted downloads after system restart. An incremental update strategy is adopted, and processing operations are only performed on new files or files whose fingerprints have changed.

7. The method according to claim 1, characterized in that, The data processing pipeline mentioned in step S4 specifically includes: Data Acquisition and Analysis: Identify file formats based on file extensions and content features, parse CSV, JSON, and XML content, and detect file encoding; Data cleaning and validation: handling missing values ​​and outliers, performing data type conversion and deduplication; Data conversion and storage: Perform format standardization, field mapping, batch writing to the database, and record the processing status.

8. The method according to claim 1, characterized in that, The method further includes: Process health monitoring and automatic recovery: Real-time monitoring of the running status of all worker processes. When a process crashes or freezes, the abnormal process is terminated and a new process is automatically restarted based on the original configuration parameters, thus achieving self-healing.

9. An adaptive file monitoring and data processing system based on a multi-process architecture, characterized in that, The system implements the steps of the adaptive file monitoring and data processing method based on a multi-process architecture as described in any one of claims 1-8 during runtime, and the system includes: Command line interface layer: used to parse command line parameters, serving as the system startup entry point; Process management layer: This includes the process manager, which is responsible for creating, monitoring, and recycling worker processes, and for implementing process health monitoring and automatic recovery; Worker process layer: contains multiple independent DirectoryWorker processes, each responsible for monitoring a directory and configuring an independent data processing pipeline; Monitoring strategy layer: includes monitoring strategy selector, watchdog event monitor and polling monitor, which dynamically selects the monitoring mode based on environmental detection results; Data processing layer: includes components for data parsing, cleaning, validation, transformation, and data storage; Communication and State Layer: Includes result queues, stop events, and file state storage, used for inter-process communication and incremental processing state tracking.

10. An adaptive file monitoring and data processing device based on a multi-process architecture, characterized in that, The device, when running, implements the steps of the adaptive file monitoring and data processing method based on a multi-process architecture as described in any one of claims 1-8, wherein the device comprises: The multi-process initialization and configuration management module is used to create a corresponding number of independent worker processes based on the number of CPU cores when the system starts up. Each worker process is responsible for monitoring a specified directory. The system configuration is loaded through a unified configuration management framework. The system configuration includes the monitoring directory, processing parameters, and database connection information. An adaptive monitoring strategy selection module is used to analyze operating environment parameters based on an environment detection algorithm and dynamically select the optimal monitoring mode. The monitoring modes include a real-time monitoring mode based on file system events and a polling mode based on time intervals. When an abnormality occurs in the monitoring mode, a dynamic switching mechanism is triggered to automatically downgrade to the polling mode and support recovery. The incremental file processing module is used to adopt an incremental file processing mechanism based on file fingerprints. By comparing the fingerprint of the new file with the historical records, it identifies newly added or modified files for processing, avoiding repeated operations on already processed files. The data processing pipeline module is used to perform data cleaning, quality verification, format conversion, and batch storage operations on the identified files, completing the entire process from data acquisition to storage.

Images