File monitoring system and file monitoring method for data warehouse cluster

Abstract

The invention discloses a file monitoring system and a file monitoring method for a data warehouse cluster, and the system comprises the steps that a file system monitoring module monitors a file change event, calculates a file hash value when a preset condition is matched, and records operation information; the mobile storage equipment monitoring module dynamically detects a plugging event of equipment such as a USB flash disk and adjusts a monitoring path of the file system monitoring module according to the plugging event; the flow monitoring module captures and analyzes a file transmission protocol data packet and records file transmission operation information; the heartbeat mechanism module periodically sends the operation state of the client to the management server; the configuration management module dynamically reloads the configuration file in the operation process so as to update parameters of each module; the log recording module records various monitoring events in a unified manner; and the network communication module sends the operation information and the heartbeat data to a management server to realize centralized management and control. According to the invention, the mobile storage device can be identified, the monitoring blind area is eliminated, the monitoring parameters are dynamically updated, and the flexibility and adaptability of the system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of file monitoring, and more particularly to a file monitoring system and a file monitoring method for a data warehouse cluster. BACKGROUND

[0002] With the deepening of enterprise digital transformation, the data warehouse cluster has become a key infrastructure for storing and analyzing core business data. The security, integrity and traceability of the internal files are directly related to the data governance compliance and business secret protection of the enterprise. Therefore, real-time and comprehensive monitoring of file operations (including local modification and network transmission) on each node in the data warehouse cluster has become an indispensable technical requirement.

[0003] At present, the common file monitoring solutions in the industry mainly fall into two categories. One is to rely on the file system event notification mechanism provided by the operating system to listen to specific directories to capture file creation, modification, deletion and other events. The other is to analyze the traffic of common file transfer protocols (such as File Transfer Protocol) through packet capture at the network level to restore file upload and download operations. These solutions are usually deployed independently, and the monitoring range is relatively fixed, and the configuration information is generally loaded when the service starts.

[0004] However, the existing technology still has some defects. First, the monitoring range is rigid, and it cannot automatically identify and dynamically include mobile storage devices such as U disks, resulting in a blind area in monitoring; second, the system flexibility is poor, and the monitoring parameter update must restart the service, which cannot adapt to the dynamic changes of the data warehouse cluster; third, the monitoring dimension is single, and the local file operation, mobile device and network transmission monitoring are mutually fragmented, lacking unified coordination. These deficiencies make the existing solutions difficult to meet the high-standard monitoring requirements of the data warehouse cluster in terms of comprehensiveness, flexibility and integrated management. SUMMARY

[0005] In view of at least one defect or improvement requirement of the prior art, the present application provides a file monitoring system and a file monitoring method for a data warehouse cluster, which solves the problems that the prior art cannot automatically identify mobile storage devices, resulting in a blind area in monitoring, the monitoring parameter update must restart the service, which cannot adapt to the dynamic changes of the data warehouse cluster, and the local file operation, mobile device and network transmission monitoring are mutually fragmented, lacking unified coordination.

[0006] To achieve the above-mentioned purpose, according to the first aspect of the present application, a file monitoring system for a data warehouse cluster is provided, comprising: a file system monitoring module for listening to file change events through a file system notification mechanism of an operating system, judging whether a preset monitoring condition is matched when a file change is detected, and calculating a hash value of the file and recording corresponding operation information when the condition is matched; a mobile storage device monitoring module for working with the file system monitoring module, periodically detecting a storage device list, identifying insertion or removal events of a mobile storage device, and dynamically adding or removing a monitoring path to the file system monitoring module according to the identification result; a traffic monitoring module for capturing a data packet of a file transfer protocol control port, parsing a file transfer protocol command and a parameter, and recording corresponding operation information; a heartbeat mechanism module for periodically sending a heartbeat packet containing a running state of the monitoring client to a management server; a configuration management module connected with the file system monitoring module, the mobile storage device monitoring module, the traffic monitoring module and the heartbeat mechanism module, for reloading a configuration file to update monitoring parameters of each module at a preset period during running; a log recording module for recording event information from the file system monitoring module, the mobile storage device monitoring module and the traffic monitoring module; a network communication module connected with the file system monitoring module, the traffic monitoring module and the heartbeat mechanism module, for transmitting operation information and the heartbeat packet to the management server.

[0007] In a possible implementation, the file system monitoring module scans a monitoring path specified by the configuration management module at initialization, calculates an initial hash value of a file and stores the hash value, and detects whether a file content changes by comparing the hash value changes.

[0008] In a possible implementation, the mobile storage device monitoring module identifies an insertion event or a removal event of a mobile storage device by monitoring changes of a storage device list, and notifies the file system monitoring module of a mounting path of the mobile storage device to start monitoring when the mobile storage device is inserted, and notifies the file system monitoring module to stop monitoring the path and release related resources when the mobile storage device is removed.

[0009] In a possible implementation, the traffic monitoring module captures and parses a file transfer protocol control connection packet and a data transmission packet, and records operation information including a username, a source network address, a target network address, a file name, a file size and an operation time.

[0010] In a possible implementation, the configuration management module dynamically updates monitoring parameters including a monitoring path, a file type filtering rule, an exclusion path, a server address, a client identifier and a heartbeat sending interval.

[0011] In a possible implementation, the log recording module assigns different log recording levels to different categories of events. When the log file volume grows to a predetermined size, the current log file is automatically archived and a new log file is created; and the maximum number of log files is set, and when the number is reached, the oldest archived log file is automatically deleted.

[0012] In a possible implementation, the network communication module adopts a message batch sending mechanism, and sends a plurality of operation information generated within a predetermined time window to the management server after merging the plurality of operation information into one data packet.

[0013] In a possible implementation, the heartbeat mechanism module is configured to: compare the current resource usage with the resource usage at the last sending time each time a heartbeat packet is constructed; If the change amount of the preset index does not exceed the preset threshold, a lightweight heartbeat packet is sent; If the change amount of the preset index exceeds the preset threshold, a complete heartbeat packet is sent.

[0014] According to a second aspect of the present application, a file monitoring method of a data warehouse cluster is also provided, based on the file monitoring system of any one of the above, comprising the following steps: Initializing the monitoring client, generating a unique identifier and loading monitoring parameters through the configuration management module; Starting the file system monitoring thread, the mobile storage device monitoring thread and the file transfer protocol traffic monitoring thread; Listening to file change events through the file system monitoring thread, calculating the file hash value and recording the corresponding operation information when detecting file operations; Detecting the system storage device list regularly through the mobile storage device monitoring thread, identifying the insertion and removal events of the mobile storage device, and dynamically adjusting the monitoring range of the file system monitoring thread accordingly; Capturing the data packet of the file transfer protocol control port through the file transfer protocol traffic monitoring thread, parsing the command and parameter, and recording the corresponding operation information; Sending a heartbeat packet containing the running state of the monitoring client to the management server regularly through the heartbeat mechanism module; Re-loading the configuration file regularly through the configuration management module to update the running parameters of each monitoring thread; Recording the operation information of various monitoring events through the log recording module; Transmitting the monitoring data to the management server through the network communication module.

[0015] In a possible implementation, the method further comprises: During the running of the monitoring client, the interruption signal of the file monitoring system is listened to in real time. When the interrupt signal is captured, the following exit procedures are executed in sequence: Send a stop instruction to the file system monitoring thread, the mobile storage device monitoring thread and the file transfer protocol traffic monitoring thread; Close the network connection with the management server, and record the program exit log information.

[0016] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects: The file monitoring system of the data warehouse cluster provided by the present application can periodically detect the storage device list through the mobile storage device monitoring module, identify the insertion or removal event of the mobile storage device, and dynamically add or remove the monitoring path to the file system monitoring module according to the identification result, so that the operation of the mobile storage device can be fully covered, and the mobile storage device can be timely included in the monitoring range regardless of when the device is connected or removed, thereby eliminating the monitoring blind area caused by the inability to identify the mobile storage device. The configuration management module reloads the configuration file at a preset period during operation to update the monitoring parameters of each module, so that the system can timely respond to the changes of the data warehouse cluster without restarting the service, ensuring that the monitoring parameters are always matched with the actual state of the cluster, and improving the flexibility and adaptability of the system. The file system monitoring module listens to the file change event, the mobile storage device monitoring module cooperates with the file system monitoring module to identify the mobile storage device operation, the traffic monitoring module captures the file transfer protocol related data packet record operation information, the log recording module uniformly records the event information from multiple modules, and the network communication module transmits the operation information to the management server, thereby realizing the unified and cooperative monitoring of the local file operation, the mobile device operation and the network transmission operation. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The structural schematic diagram of an embodiment of the file monitoring system of the data warehouse cluster provided by the present application is shown in the figure. Figure 2 The flowchart of an embodiment of the file monitoring method of the data warehouse cluster provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0020] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0021] This invention provides a file monitoring system and method for a data warehouse cluster, which will be described below.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram of an embodiment of the file monitoring system for a data warehouse cluster provided by the present invention. In a specific embodiment of the present invention, a file monitoring system 100 for a data warehouse cluster is disclosed, comprising: The file system monitoring module 110 is used to listen for file change events through the file system notification mechanism of the operating system. When a file change is detected, it determines whether it matches the preset monitoring conditions, and calculates the hash value of the file and records the corresponding operation information when it matches. The mobile storage device monitoring module 120 is used to work in conjunction with the file system monitoring module 110 to periodically check the storage device list, identify the insertion or removal events of mobile storage devices, and dynamically add or remove monitoring paths to the file system monitoring module 110 based on the identification results. The traffic monitoring module 130 is used to capture data packets on the file transfer protocol control port, parse file transfer protocol commands and parameters, and record corresponding operation information; The heartbeat mechanism module 140 is used to periodically send heartbeat packets containing the monitoring client's running status to the management server 200; The configuration management module 150 is connected to the file system monitoring module 110, the mobile storage device monitoring module 120, the traffic monitoring module 130 and the heartbeat mechanism module 140 respectively, and is used to reload the configuration file at a preset period during operation to update the monitoring parameters of each module. The log recording module 160 is configured to record event information from the file system monitoring module 110, the mobile storage device monitoring module 120 and the traffic monitoring module 130. The network communication module 170 is connected with the file system monitoring module 110, the traffic monitoring module 130 and the heartbeat mechanism module 140, and is configured to transmit operation information and heartbeat packets to the management server 200.

[0023] In the above embodiment, the file system monitoring module 110 continuously monitors the preset monitoring path by calling the system native file system notification mechanism, and can not only capture the creation, modification, deletion and movement events of files, but also verify the content integrity. Specifically, when a file operation is detected and the path and type thereof meet the monitoring conditions, the cryptographic hash value of the file is calculated in real time, the change of the hash value is compared to determine whether the file content is tampered, and the specific operation data is recorded.

[0024] The mobile storage device monitoring module 120 cooperates with the file system monitoring module 110 to solve the monitoring problem of temporary storage devices. The module dynamically tracks the change of the storage device list by polling the system device interface or listening to the device management event, and identifies the insertion and removal of mobile media such as U disks. Once a new device is mounted, the path thereof is immediately added to the file system monitoring range; when the device is removed, the corresponding monitoring task is automatically cleaned up and the resource is released, and the elasticity and self-adaptation of the monitoring range are realized.

[0025] The traffic monitoring module 130 captures the data stream flowing through the file transfer protocol control port (generally defaulting to 21) through network packet capturing technology, deeply analyzes the file transfer protocol command sequence (such as USER, PASS, RETR, STOR, CWD, etc.), restores the user's login, authentication, directory switching and file transfer behavior, and extracts key information such as username, source / destination IP, file name, file size and operation timestamp, to form a complete operation audit log.

[0026] The heartbeat mechanism module 140 is responsible for maintaining the communication link and state visibility between the monitoring client and the management server 200. The heartbeat data packet is sent to the management server 200 at a configurable time interval, and the packet encapsulates the client identifier, system resource occupation (such as CPU, memory usage) and monitoring thread health status, etc. Key operating indicators, so that the management end can sense the node survival state in real time and perform unified coordination.

[0027] The configuration management module 150 serves as the control center of the system and provides parameter services for the above-mentioned functional modules through a loosely coupled connection mode. It supports dynamic reloading of external configuration files (usually in structured formats such as JSON or XML) at a preset period without interrupting the operation of the system, realizes real-time hot updating of parameters such as monitoring paths, file type filtering rules, exclusion lists, server addresses, and heartbeat intervals, and greatly improves the flexibility and maintainability of the system.

[0028] The log recording module 160 receives and aggregates event information from various modules such as the file system, mobile devices, and network traffic, and records operation events, system abnormalities, and state changes to local log files in a predefined format and level, providing tamper-proof data for security audits, troubleshooting, and behavior analysis for subsequent data queries.

[0029] The network communication module 170 is responsible for transmitting monitoring data such as file operation events, file transfer protocol operation logs, and heartbeat packets to the central management server 200 through efficient network protocols (such as HTTP / HTTPS or custom TCP protocols). This module usually uses asynchronous sending, data batch merging and compression optimization strategies to ensure timely reporting while minimizing the impact on local performance and network bandwidth.

[0030] Compared with the prior art, the file monitoring system 100 of the data warehouse cluster provided by the embodiment can periodically detect the storage device list through the mobile storage device monitoring module 120, identify the insertion or removal event of the mobile storage device, and dynamically add or remove the monitoring path to the file system monitoring module 110 according to the identification result, which can comprehensively cover the operation of the mobile storage device and eliminate the monitoring blind area caused by the inability to identify the mobile storage device. The configuration management module 150 reloads the configuration file at a preset period during operation to update the monitoring parameters of each module, which can respond to changes in the data warehouse cluster in time without restarting the service, ensuring that the monitoring parameters always match the actual state of the cluster, improving the flexibility and adaptability of the system. The file system monitoring module 110 listens to file change events, the mobile storage device monitoring module 120 cooperates with the file system monitoring module 110 to identify mobile storage device operations, the traffic monitoring module 130 captures file transfer protocol-related data packets to record operation information, the log recording module 160 uniformly records event information from multiple modules, and the network communication module 170 transmits operation information to the management server 200, realizing unified and cooperative monitoring of local file operations, mobile device operations, and network transmission operations.

[0031] In some embodiments of the present application, the file system monitoring module 110 scans the monitoring paths specified by the configuration management module 150 at initialization, calculates the initial hash value of the file and stores it, and detects whether the file content has changed by comparing the hash value changes.

[0032] In the above embodiment, the file system monitoring module 110 first reads one or more monitoring paths specified by the configuration management module 150 during the initialization stage, and performs a recursive scan on all target files that meet the conditions under these paths. In this process, a unique initial hash value is calculated for each file using a cryptographic hash algorithm such as MD5 or SHA-256, and the mapping relationship between this hash value and the file path is stored in the local database, establishing an initial file content baseline snapshot. In the subsequent real-time monitoring process, when a file modification event is detected, the current hash value of the file is recalculated and compared with the initial hash value stored in the database. If the comparison result is inconsistent, it is confirmed that the file content has changed substantially, and the operation record and reporting process is triggered, which not only captures the metadata changes of the file, but also accurately identifies the changes in the file content itself, effectively improving the accuracy of the monitoring.

[0033] In some embodiments of the present application, the mobile storage device monitoring module 120 identifies the insertion event or removal event of the mobile storage device by monitoring the changes in the storage device list, and notifies the file system monitoring module 110 of the mounting path of the mobile storage device when it is inserted to start monitoring; when the mobile storage device is removed, it notifies the file system monitoring module 110 to stop monitoring the path and release related resources.

[0034] In the above embodiment, the mobile storage device monitoring module 120 continuously polls the storage device list interface provided by the operating system kernel or listens to the device management events at the system bottom layer to realize real-time sensing of the dynamic changes of the storage device. When a new mobile storage device (such as a USB flash disk or a mobile hard disk) is inserted into the system and successfully mounted, the accessible mounting path allocated by the operating system is immediately parsed, and this path is actively notified to the file system monitoring module 110 as a new monitoring target. The file system monitoring module 110 initializes the monitoring context for the path and includes it in the real-time event listening range, without any manual intervention or system restart. Conversely, when the device is removed, its removal is quickly detected, and the file system monitoring module 110 is notified again. Upon receiving the notification, the file system monitoring module 110 will immediately stop listening to the path and clean up all memory data structures and hash records related to it, completely releasing system resources, thereby realizing dynamic and lossless synchronization of the monitoring range and the physical device state.

[0035] In some embodiments of the present application, the traffic monitoring module 130 captures and parses file transfer protocol control connection packets and data transfer packets, and records operation information including username, source network address, target network address, file name, file size, and operation time.

[0036] In the above embodiments, the traffic monitoring module 130 first filters out data streams with target port as file transfer protocol control port (usually 21), parses packets in the control connection, and extracts key file transfer protocol commands and their response codes. Through comprehensive analysis of these information, complete operation audit records are constructed, including but not limited to: username of the operation, source IP address and port of the connection initiation, target server IP address of the connection reception, file name being transferred, file size, and time stamp accurate to milliseconds, thereby achieving comprehensive and fine-grained monitoring of file transfer protocol file transfer activities.

[0037] In some embodiments of the present application, the configuration management module 150 dynamically updates monitoring parameters including monitoring path, file type filtering rule, exclusion path, server address, client identifier, and heartbeat sending interval.

[0038] In the above embodiments, the configuration management module 150 supports a series of monitoring parameters that are dynamically updated at runtime, including: monitoring path, i.e. a set of file system directories that need to be monitored; file type filtering rule, based on wildcard or regular expression, used to specify only monitoring files with specific extensions; exclusion path, used to ignore certain known subdirectories or temporary files that do not need to be monitored; server address, i.e. the network endpoint of the management server 200, used for data reporting and instruction receiving; client identifier, as the unique identity ID of each monitoring node in the cluster; and heartbeat sending interval, used to control the frequency of status information reporting. All parameters are stored in a structured configuration file, and by periodically checking and reloading the file, the update of all monitoring strategies is achieved without interrupting the running monitoring service.

[0039] In some embodiments of the present application, the log recording module 160 assigns different log recording levels to different categories of events; When the log file volume grows to a predetermined size, the current log file is automatically archived and a new log file is created; and the maximum number of log files is set, and when this number is reached, the oldest archived log file is automatically deleted.

[0040] In the above embodiments, the log recording module 160 introduces a log level mechanism, which assigns different log levels to events of different categories and importance, and filters out unimportant information according to the current running mode. It also continuously monitors the size of the current active log file, and when the size grows to a predetermined threshold (e.g., 10 MB), it renames the current file and archives it (e.g., appends a timestamp), while creating a brand new empty log file to continue recording, preventing the single log file from growing indefinitely. It also sets a maximum number of log files (e.g., keeping the last 20 archived files), and when the number of archived files exceeds this limit, it deletes the archived file with the earliest creation time, thus achieving self-management of log storage space and avoiding disk space exhaustion.

[0041] In some embodiments of the present application, the network communication module 170 adopts a message batch sending mechanism, which combines multiple operation information generated within a predetermined time window into a data packet and then sends it to the management server 200.

[0042] In the above embodiments, the network communication module 170 adopts an intelligent message batch sending mechanism to optimize network utilization efficiency and reduce the receiving pressure of the management server 200. It maintains a first-in-first-out sending cache area in the module. All operation information to be sent generated by the file system monitoring module 110 and the traffic monitoring module 130 is not immediately sent out, but first stored in this cache area. A time window (e.g., 5 seconds) and a quantity threshold (e.g., 100 messages) are set. As long as either of the two conditions, i.e., the time window expires or the number of cached messages reaches the threshold, is met, the network communication module 170 will trigger a sending operation, package all the operation information accumulated in the cache area into a single, structured data packet, and batch send it to the management server 200. This reduces the network overhead and server connection number caused by high frequency and small data packets, and improves the overall system throughput.

[0043] In some embodiments of the present application, the heartbeat mechanism module 140 is configured to: compare the current resource usage with the resource usage at the last sending time when constructing the heartbeat packet each time; If the change amount of the preset index does not exceed the preset threshold, a lightweight heartbeat packet is sent; If the change amount of the preset index exceeds the preset threshold, a complete heartbeat packet is sent.

[0044] In the above embodiment, before constructing the heartbeat packet each time, the heartbeat mechanism module 140 collects the key indicators of the current system resource usage of the monitoring client, such as CPU occupancy, memory usage, disk I / O, etc., and compares these values with the historical values recorded at the last time when the heartbeat packet was successfully sent, and calculates the change amount. If the change amount of all key indicators does not exceed the respective threshold values, it indicates that the system state is stable, and at this time only a lightweight heartbeat packet is sent, which only contains the client identifier and the basic state flag, and the size is very small. Otherwise, if the change amount of any one or more indicators exceeds the preset threshold value, it indicates that the system state has undergone significant fluctuations, and at this time the module sends a complete heartbeat packet, which contains not only the aforementioned basic identification but also rich information such as detailed resource usage data and health state snapshot of each monitoring thread.

[0045] Referring to Figure 2 , Figure 2 An embodiment of the file monitoring method of the data warehouse cluster provided by the present application is shown in the flowchart. According to the second aspect of the present application, a file monitoring method of a data warehouse cluster is also provided, which is based on the file monitoring system 100 of any one of the above embodiments, and includes the following steps: S201, initializing the monitoring client, generating a unique identifier and loading the monitoring parameters through the configuration management module; S202, starting the file system monitoring thread, the mobile storage device monitoring thread and the file transfer protocol traffic monitoring thread; S203, listening to file change events through the file system monitoring thread, calculating the file hash value and recording the corresponding operation information when detecting file operations; S204, periodically detecting the system storage device list through the mobile storage device monitoring thread, identifying the insertion and removal events of the mobile storage device, and dynamically adjusting the monitoring range of the file system monitoring thread accordingly; S205, capturing the data packet of the file transfer protocol control port through the file transfer protocol traffic monitoring thread, parsing the command and parameter, and recording the corresponding operation information; S206, periodically sending the heartbeat packet containing the running state of the monitoring client to the management server through the heartbeat mechanism module; S207, periodically reloading the configuration file through the configuration management module to update the running parameters of each monitoring thread; S208, recording the operation information of various monitoring events through the log recording module; S209, transmitting the monitoring data to the management server through the network communication module.

[0046] In the above embodiment, the monitoring client is deployed and started on the target data warehouse cluster node, first a globally unique identifier is generated, which is bound with the node hardware information, and stored in the local configuration file. Subsequently, the initial monitoring parameters are loaded through the configuration management module 150, including the monitoring path, file type filtering rules, server address and heartbeat interval, etc., to complete the construction of the running environment.

[0047] After initialization, three core monitoring threads are started in parallel: the file system monitoring thread, responsible for real-time monitoring of local file system events; the mobile storage device monitoring thread, responsible for dynamically sensing the insertion and removal of USB and other mobile storage devices; the file transfer protocol traffic monitoring thread, responsible for capturing and analyzing file transfer protocol traffic in the network. Each thread runs as an independent concurrent unit, ensuring full coverage of monitoring dimensions and mutual non-blocking.

[0048] Through the file system monitoring thread, the file creation, modification, movement and deletion events under the specified path are continuously monitored. When a file operation is detected, it is judged whether it matches the preset monitoring conditions. For files that match the conditions, the current content hash value is immediately calculated, compared with the pre-stored reference hash value, and the file content is judged whether it has undergone substantial changes. The complete operation information including operation type, file path, timestamp and hash value is recorded.

[0049] Through the mobile storage device monitoring thread, the system storage device list is scanned periodically at a configurable time interval, or the device change events issued by the operating system are listened to. When a new mobile storage device is recognized to be inserted, its mounting path is parsed, and the path is dynamically added to the monitoring list of the file system monitoring thread, realizing plug-and-play monitoring. When the device is removed, the corresponding path is immediately removed from the monitoring list, and the file system monitoring thread is notified to clean up related resources, ensuring that the monitoring range and the physical device state are real-time synchronized.

[0050] Through the file transfer protocol traffic monitoring thread, the original data packet of the file transfer protocol control port is captured at the network interface layer, the data packet is reassembled and deeply analyzed, and the key commands and parameters in the file transfer protocol session are extracted. By analyzing the USER, PASS, RETR, STOR, etc. Command sequence, the user login, file upload, file download, etc. Operation is restored, and the key audit information such as username, source IP, target IP, file name, file size and operation timestamp is recorded.

[0051] The heartbeat mechanism module 140 periodically sends heartbeat data packets to the management server 200 at the time interval set by the configuration management module 150. The heartbeat packet encapsulates the unique identifier of the client, the current system resource usage, and the running health status of each monitoring thread, enabling the management server 200 to real-time monitor the survival status and load of all monitoring nodes, and achieving unified management of the cluster.

[0052] The configuration management module 150 periodically checks and reloads the external configuration file in the main program loop, enabling hot updating of monitoring parameters without restarting the monitoring service, including dynamically adjusting monitoring paths, updating file filtering rules, modifying heartbeat frequency, etc., enabling the system to flexibly adapt to dynamic changes in data warehouse cluster size and business requirements.

[0053] The log recording module 160 receives and aggregates event and status information from each monitoring thread and functional module, and writes key information such as file change events, device plug-in events, file transfer protocol operations, heartbeat sending results, and configuration loading results into local log files according to the preset log level and format, providing complete traceability for system operation and maintenance, security audit, and fault diagnosis.

[0054] The network communication module 170 aggregates, serializes, and encapsulates the operation information generated by the file system monitoring thread and the file transfer protocol traffic monitoring thread, as well as the status data generated by the heartbeat mechanism module 140. Using batch sending and asynchronous transmission mechanisms, the monitoring data is efficiently and reliably transmitted to the central management server 200, completing centralized collection and reporting of data.

[0055] In some embodiments of the present application, it further includes: During the operation of the monitoring client, the interruption signal of the file monitoring system is real-time monitored; When the interruption signal is captured, the following exit procedures are executed in order: Stop instructions are sent to the file system monitoring thread, the mobile storage device monitoring thread, and the file transfer protocol traffic monitoring thread; The network connection with the management server 200 is closed, and program exit log information is recorded.

[0056] In the above embodiment, during the operation of the main process of the monitoring client, a signal processing function is registered with the operating system to real-time monitor and capture the preset system interruption signal, including the keyboard interruption signal initiated by the user and the termination signal required by the system, thereby providing a controllable exit entry for the program.

[0057] When the above interrupt signal is captured, it does not immediately force exit, but first starts a thread-safe termination sequence, and sends a stop command to the file system monitoring thread, the mobile storage device monitoring thread and the file transfer protocol traffic monitoring thread in sequence. After each thread receives the command, it will immediately stop receiving new events, and continue to process the current task already in the queue, or wait for a very short timeout to ensure that the current operation is completed, and then clean up the memory, handles and other resources allocated in the thread, and finally safely end the operation, avoiding the data loss or state inconsistency problems that may be caused by thread forced termination.

[0058] After all monitoring threads are confirmed to stop, all established network connections with the management server 200 are closed through the network communication module 170. Before closing, it can attempt to send the monitoring data in the sending buffer that has not been transmitted to ensure that the last batch of operation information is not lost. After the connection is closed, the network port, socket and other resources occupied are immediately released.

[0059] Before the program is finally terminated, a detailed program exit log is recorded by the log recording module 160, which at least contains the exit reason (such as the specific signal type captured), the accurate timestamp, and the session identifier of this run.

[0060] It should be noted that for the foregoing method embodiments, in order to simply describe, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0061] In the above embodiments, the description of each embodiment is focused on, and the part not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0062] In the several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner for actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical or other forms.

[0063] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may also be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0064] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0065] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: a U disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0066] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable memory, which can include a flash disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0067] The above is only an exemplary embodiment of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will easily think of embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present application is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional techniques in the art that are not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

[0068] Any combination of the technical features of the above embodiments can be made, and for the sake of brevity, not all possible combinations of the technical features of the above embodiments are described, however, any combination of the technical features should be considered as within the scope of the present disclosure, as long as the combination is not contradictory.

[0069] Those skilled in the art easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A file monitoring system for a data warehouse cluster, characterized in that, include: The file system monitoring module is used to listen for file change events through the operating system's file system notification mechanism. When a file change is detected, it determines whether it matches the preset monitoring conditions, and if it matches, it calculates the file's hash value and records the corresponding operation information. The mobile storage device monitoring module is used to work in conjunction with the file system monitoring module to periodically check the storage device list, identify insertion or removal events of mobile storage devices, and dynamically add or remove monitoring paths to the file system monitoring module based on the identification results. The traffic monitoring module is used to capture data packets on the file transfer protocol control port, parse file transfer protocol commands and parameters, and record corresponding operation information; The heartbeat mechanism module is used to periodically send heartbeat packets containing monitoring client running status to the management server; The configuration management module is connected to the file system monitoring module, the mobile storage device monitoring module, the traffic monitoring module and the heartbeat mechanism module respectively, and is used to reload the configuration file at a preset period during operation to update the monitoring parameters of each module. The log recording module is used to record event information from the file system monitoring module, the mobile storage device monitoring module, and the traffic monitoring module; The network communication module is connected to the file system monitoring module, the traffic monitoring module, and the heartbeat mechanism module, and is used to transmit the operation information and the heartbeat packet to the management server.

2. The file monitoring system for a data warehouse cluster according to claim 1, characterized in that, During initialization, the file system monitoring module scans the monitoring path specified by the configuration management module, calculates and stores the initial hash value of the file, and detects whether the file content has changed by comparing the changes in the hash value.

3. The file monitoring system for a data warehouse cluster according to claim 1, characterized in that, The mobile storage device monitoring module identifies insertion or removal events of mobile storage devices by monitoring changes in the storage device list. When a mobile storage device is inserted, it notifies the file system monitoring module of its mount path to start monitoring. When a mobile storage device is removed, it notifies the file system monitoring module to stop monitoring the path and release related resources.

4. The file monitoring system for a data warehouse cluster according to claim 1, characterized in that, The traffic monitoring module captures and parses file transfer protocol control connection packets and data transmission packets, and records operation information including username, source network address, destination network address, file name, file size, and operation time.

5. The file monitoring system for a data warehouse cluster according to claim 1, characterized in that, The monitoring parameters dynamically updated by the configuration management module include monitoring path, file type filtering rules, exclusion paths, server address, client identifier, and heartbeat sending interval.

6. The file monitoring system for a data warehouse cluster according to claim 5, characterized in that, The logging module assigns different logging levels to different categories of events; When the log file size grows to the predetermined size, the current log file is automatically archived and a new log file is created; and a maximum number of log files is set, and the oldest archived log file is automatically deleted when this number is reached.

7. The file monitoring system for a data warehouse cluster according to claim 1, characterized in that, The network communication module adopts a message batch sending mechanism, which merges multiple operation information generated within a predetermined time window into a single data packet and sends it to the management server.

8. The file monitoring system for a data warehouse cluster according to claim 1, characterized in that, The heartbeat mechanism module is configured to compare the current resource utilization rate with the resource utilization rate during the previous transmission each time a heartbeat packet is constructed. If the change in the preset indicator does not exceed the preset threshold, a lightweight heartbeat packet is sent. If the change in the preset indicator exceeds the preset threshold, a complete heartbeat packet will be sent.

9. A file monitoring method for a data warehouse cluster, based on the file monitoring system for a data warehouse cluster as described in any one of claims 1 to 8, characterized in that, Includes the following steps: Initialize the monitoring client, generate a unique identifier, and load monitoring parameters through the configuration management module; Start the file system monitoring thread, the removable storage device monitoring thread, and the file transfer protocol traffic monitoring thread; The file system monitoring thread listens for file change events, calculates the file hash value and records the corresponding operation information when a file operation is detected; The system's storage device list is periodically checked by the removable storage device monitoring thread to identify insertion and removal events of removable storage devices, and the monitoring scope of the file system monitoring thread is dynamically adjusted accordingly. The file transfer protocol traffic monitoring thread captures data packets whose target port is the file transfer protocol control port, parses commands and parameters, and records corresponding operation information. The heartbeat mechanism module periodically sends heartbeat packets containing monitoring client running status to the management server. The configuration management module periodically reloads the configuration file to update the running parameters of each monitoring thread; The logging module records operational information for various monitored events. The monitoring data is transmitted to the management server via the network communication module.

10. The file monitoring method for a data warehouse cluster according to claim 9, characterized in that, Also includes: During the operation of the monitoring client, interrupt signals of the file monitoring system are monitored in real time. When the interrupt signal is detected, the following exit procedure is executed in sequence: Send a stop command to the file system monitoring thread, the removable storage device monitoring thread, and the file transfer protocol traffic monitoring thread; Close the network connection with the management server and record the program exit log information.

Images