Intelligent server monitoring method for FTP (File Transfer Protocol) file

Abstract

The invention relates to the technical field of FTP file management, in particular to a server intelligent monitoring method for FTP files. The method comprises the following steps: performing snapshot processing on a target folder in an FTP file server, storing initial snapshot information to a monitoring manager, generating a monitoring process and processing the target folder to obtain changed snapshot information after monitoring that a file operation command is executed on the target folder, the initial snapshot information and the change snapshot information are compared to determine whether the file in the target folder is changed or not, and the file information is imported into the data warehouse through the monitoring process after the change, so that the pressure of the FTP file server is reduced; the data transmission duration of the initial snapshot information obtained by the monitoring process is counted, the duration mean value is obtained based on the data transmission durations, then the operation process of the monitoring manager is judged, a corresponding instruction is generated when the monitoring manager is unqualified, corresponding parameters are determined and adjusted, and the data transmission duration is shortened, so that the efficiency of importing the changed file into the data warehouse is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of FTP file management, and in particular to an intelligent monitoring method for FTP files on a server. Background Art

[0002] In big data processing, the File Transfer Protocol (FTP), as a traditional file transfer method, still occupies an important position in the construction of a data warehouse. During the data import process, it is often necessary to read specific data files from an FTP file server. However, due to network latency, file generation latency, transmission errors, or other unforeseen reasons, the target file to be read may not exist or may not be ready yet, which poses a great challenge to the data import process. Currently, for monitoring file changes on an FTP file server, methods such as timed polling or real-time monitoring are usually adopted. Timed polling is to continuously query the file list on the FTP server at a set fixed time interval to determine whether the target file exists or has changed. However, this method has obvious drawbacks: on the one hand, if the set polling interval is too short (such as monitoring every second), it will cause too much pressure on the FTP server, resulting in a decline in server performance and even affecting other normal file transfer operations; on the other hand, if the polling interval is too long, it may lead to insufficient timeliness of data import and fail to meet the business requirements for data real-time. Although real-time monitoring technology can solve the timeliness problem to a certain extent, it also faces the problem of resource waste.

[0003] The Chinese patent publication number: CN1997019B in the prior art provides a message monitoring and receiving method based on FTP transmission. In this technical solution, an FTP client sends a file to an FTP server, and the information and content in the file are analyzed in real time to determine whether they meet a predetermined standard. The files determined to meet the predetermined standard are sent to a codec module. The codec module decodes the files sent by the FTP server and analyzes the information contained in the files. The display storage module displays the file information in real time according to the results analyzed by the codec module and stores the files in a database, thereby timely understanding the situation of the monitored user. By formatting and storing in the database, the speed and efficiency of subsequent message analysis, query, and processing are improved. This technical solution also determines subsequent operations by analyzing files in real time, which will cause excessive operating pressure on the FTP server and does not involve how to optimize the data transmission process based on the data transmission duration, resulting in low data transmission efficiency. Summary of the Invention

[0004] To this end, the present invention provides a server intelligent monitoring method for FTP files to solve the problem in the prior art that the data transmission duration based on snapshot information is not considered to determine the operation process of the monitoring manager and adjust corresponding parameters based on the determination result, resulting in low efficiency of importing changed file information into the data warehouse.

[0005] To achieve the above object, the present invention provides a server intelligent monitoring method for FTP files, including: Performing initial snapshot processing on several target folders in the FTP file server to obtain corresponding initial snapshot information and storing it in the monitoring manager; Connecting the FTP file server to the client, real-time monitoring the user's login event through the log system in the FTP file server, and after monitoring the user's login, obtaining the file operation commands executed by the user through the log system; Generating a monitoring process based on the execution of the file operation command in the target folder, and performing changed snapshot processing on the target folder to obtain corresponding changed snapshot information; The monitoring process obtains the corresponding initial snapshot information from the monitoring manager and compares it with the changed snapshot information to determine whether a file change has occurred in the corresponding target folder. After determining that a file change has occurred, the changed file information is imported into the data warehouse; Statistical the duration from the issuance of the acquisition request for the corresponding initial snapshot information by a single monitoring process to the receipt of the initial snapshot information sent by the monitoring manager and record it as the data transmission duration, and determine whether the operation process of the monitoring manager is qualified based on the data transmission durations corresponding to each monitoring process obtained within a preset operation cycle; When the determination is unqualified, determining corresponding instructions based on each data transmission duration, determining to increase the operating memory of the monitoring manager based on the instructions, merging duplicate monitoring processes, updating the initial snapshot information in the monitoring manager, or optimizing the transmission environment.

[0006] Further, the process of determining whether the operation process of the monitoring manager is qualified based on the data transmission durations corresponding to each monitoring process obtained within the preset operation cycle includes: Statistical each data transmission duration and calculating the average value to obtain the duration average value; Making a determination based on the comparison result between the duration average value and the preset duration average value, or re-determining whether the operation process of the monitoring manager is qualified based on the total number of monitoring processes generated within the preset operation cycle; When the determination is unqualified, determining the reason for the unqualified based on the difference between the duration average value and the preset duration average value.

[0007] Further, the process of re-determining whether the running process of the monitoring manager is qualified based on the total number of the monitoring processes generated within the preset running period includes: Count the total number of the monitoring processes and record it as the total process number; Determine whether to increase the running memory of the monitoring manager based on the comparison result between the total process number and the critical total process number.

[0008] Further, the process of determining whether to increase the running memory of the monitoring manager based on the comparison result between the total process number and the critical total process number includes: Count the total process numbers within each historical running period, and calculate the variance based on each total process number to obtain the process variance; Determine to increase the running memory based on the comparison result between the process variance and the preset process variance, and the increase amplitude of the running memory has a negative correlation with the process variance.

[0009] Further, the process of determining the reason for non-conformance based on the difference between the mean duration and the preset mean duration includes: Calculate the difference between the mean duration and the preset mean duration and record it as the mean duration difference; Determine the reason for non-conformance based on the comparison result between the mean duration difference and the preset mean duration difference; Generate a corresponding instruction based on the reason, and determine to merge the repeated monitoring processes, update the initial snapshot information in the monitoring manager, or optimize the transmission environment based on the instruction.

[0010] Further, the process of determining to merge the repeated monitoring processes includes: Count the total number of the initial snapshot information in the monitoring manager within the preset running period and record it as the total snapshot information number; Determine the merging criterion based on the comparison result between the total snapshot information number and the preset total snapshot information number, and the merging criterion has a negative correlation with the total snapshot information number; Wherein, when the number of the repeated monitoring processes is greater than or equal to the merging criterion, merge the monitoring processes.

[0011] Further, the process of determining to optimize the transmission environment includes: Count the number of the initial snapshot information simultaneously sent within the monitoring manager per unit time and record it as the snapshot concurrency number; Determine to increase the compression ratio of the initial snapshot information based on the comparison result between the snapshot concurrency number and the preset snapshot concurrency number, and the increase amplitude of the compression ratio has a positive correlation with the snapshot concurrency number.

[0012] Further, after completing the comparison between the initial snapshot information and the changed snapshot information, the monitoring process stores the changed snapshot information in the monitoring manager and replaces it with new initial snapshot information as the object for the next comparison.

[0013] Further, when the log system captures that multiple file operation commands operate on each target folder, corresponding monitoring processes are generated, and each monitoring process is connected to the monitoring manager; Obtain the paths where each target folder for operation is located, and send each path to the monitoring manager, and the monitoring manager sends the initial snapshot information corresponding to each path to the corresponding monitoring process respectively.

[0014] Further, the process of importing the changed file information into the data warehouse includes: The monitoring manager triggers an asynchronous notification mechanism and notifies the monitoring process, and the monitoring process imports the changed file information into the data warehouse; Among them, the asynchronous notification mechanism is a preset notification rule, including a message queue or a network hook.

[0015] Compared with the prior art, the beneficial effect of an intelligent monitoring method for an FTP file server of the present invention is that the method obtains initial snapshot information by performing snapshot processing on several target folders in the FTP file server and stores it in the monitoring manager. After monitoring that a user executes a file operation command on a target folder, a monitoring process is generated and snapshot processing is performed on the target folder to obtain changed snapshot information. The initial snapshot information is compared with the changed snapshot information to determine whether the files in the corresponding target folder have changed. After determining that a change has occurred, the changed file information is imported into the data warehouse through the monitoring process. The method monitors the user's file operation commands to determine whether to import the changed file information into the data warehouse, thereby reducing the pressure on the FTP file server; at the same time, the time period from when the monitoring process sends a request to obtain the corresponding initial snapshot information to when the monitoring manager sends the initial snapshot information is counted and recorded as the data transmission time period. The average value of the time periods is obtained through several data transmission time periods within a preset operation cycle. Based on the average value, it is determined whether the operation process of the monitoring manager is qualified. When it is determined that it is unqualified, corresponding instructions can be generated and corresponding parameters can be adjusted based on the instructions, thereby shortening each data transmission time period, and thus improving the efficiency and accuracy of importing files in the FTP file server into the data warehouse.

[0016] Further, based on the comparison result between the duration mean value and the preset duration mean value to determine whether the operation process of the monitoring manager is qualified, the present invention further obtains the total number of monitoring processes generated within a preset operation cycle, and then obtains the total number of processes. Based on the comparison result between the total number of processes and the critical total number of processes, the operation process of the monitoring manager is re-determined, thereby improving the accuracy of the determination process.

[0017] Further, when the present invention determines to increase the operation memory in the monitoring manager based on the comparison result between the total number of processes and the critical total number of processes, the process variance is calculated by obtaining the total number of processes in several historical operation cycles. Based on the comparison between the process variance and the preset process variance, the increase amplitude of the operation memory is determined, and then the duration of each data transmission can be shortened, thereby improving the efficiency of importing the changed file information from the FTP file server to the data warehouse.

[0018] Further, the present invention also determines the reason why the operation process of the monitoring manager is unqualified by obtaining the duration mean difference and based on the comparison result between the duration mean difference and the preset duration mean difference. Then, corresponding processing can be carried out targeted. When it is determined that there are duplicate monitoring processes, they need to be merged. When it is determined that there is a problem with the transmission environment, the compression ratio of the initial snapshot information needs to be increased. When it is determined that the corresponding initial snapshot information is missing in the monitoring manager, several target folders in the FTP file server are traversed to re-obtain new initial snapshot information and replace and store it in the monitoring manager, thereby shortening the duration of each data transmission and improving the import efficiency of the changed file information.

[0019] Further, the present invention also determines the merging criteria for merging duplicate monitoring processes based on the comparison result between the total number of snapshot information and the preset total number of snapshot information, and then targeted to shorten the duration of each data transmission, thereby improving the import efficiency of the changed file information.

[0020] Further, the present invention also determines the amplitude of increasing the compression ratio of the initial snapshot information based on the comparison result between the snapshot concurrency and the preset snapshot concurrency, and then targeted to shorten the duration of each data transmission, thereby improving the import efficiency of the changed file information. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the modules of the intelligent monitoring system for the FTP file server in the present invention; Figure 2 It is a schematic flowchart of the intelligent monitoring method for the FTP file server in the present invention; Figure 3 It is a logical decision diagram for determining whether the operation process of the monitoring manager is qualified based on the duration mean value and the corresponding processing in the present invention; Figure 4 This is a logic decision diagram for determining the reasons for the non - compliance of the monitoring manager and the corresponding handling based on the mean duration difference in the present invention. Detailed implementation manners

[0022] In order to make the objectives and advantages of the present invention more clear, the present invention will be further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] The preferred implementation manners of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these implementation manners are only used to explain the technical principles of the present invention and do not limit the protection scope of the present invention.

[0024] It should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0025] Please refer to Figure 1As shown in the figure, it is a schematic diagram of the modules of the server intelligent monitoring system for FTP files in this embodiment. It includes: an FTP file server, a monitoring manager, an initial snapshot module, a monitoring process module, a data import module, a statistics module, an analysis module, an instruction generation module, and an adjustment module.The initial snapshot module is respectively connected to the FTP file server and the monitoring manager, and is used to perform initial snapshot processing on several target folders in the FTP file server to obtain corresponding initial snapshot information, and store the initial snapshot information in the monitoring manager. Among them, the initial snapshot processing is the first static record of the target folder. The log system in the FTP file server is used to monitor users' login events in real time. After detecting a user login, the file operation commands executed by the user are captured and recorded through the log system. The monitoring process module is respectively connected to the FTP file server and the monitoring manager, and is used to generate corresponding monitoring processes when it is determined that file operation commands have been executed on several target folders. Each monitoring process is used to perform change snapshot processing on the corresponding target folder to obtain corresponding change snapshot information, and is used to obtain the initial snapshot information of the corresponding target folder from the monitoring manager, and compare based on the initial snapshot information and the change snapshot information. Among them, the change snapshot processing is a record of the modification or update of the target file on which the file operation command is executed. The data import module is connected to the monitoring process module, and is used to import the changed file information into the data warehouse when it is determined that the files in the corresponding target folder have changed. The statistics module is respectively connected to the monitoring process module and the monitoring manager, and is used to count the data transfer durations corresponding to each monitoring process module obtained within a preset operation cycle. Among them, the data transfer duration is the duration from when a single monitoring process module sends a request for obtaining the corresponding initial snapshot information to when it receives the initial snapshot information sent by the monitoring manager. The analysis module is connected to the statistics module, and is used to determine whether the operation process of the monitoring manager is qualified based on the obtained data transfer durations. When it is determined that the operation process is unqualified, the reasons are determined. The reasons include that duplicate monitoring processes occur, resulting in the simultaneous scheduling of a single initial snapshot information, which increases the data transfer duration; or the number of monitoring processes is too large, resulting in insufficient operating memory in the monitoring manager, which increases the data transfer duration; or the corresponding initial snapshot information is missing in the monitoring manager, resulting in the request for obtaining the initial snapshot information sent by the monitoring process not being responded to, which increases the data transfer duration; or there are problems in the transmission environment, which increases the data transfer duration. The instruction generation module is connected to the analysis module, and is used to generate corresponding instructions based on the reasons. The adjustment module is respectively connected to the instruction generation module, the monitoring manager, and the monitoring process module, and is used to determine to increase the operating memory of the monitoring manager based on the instructions, determine to merge the duplicate monitoring processes, determine to update the initial snapshot information in the monitoring manager, or determine to optimize the transmission environment.

[0026] Please refer to Figure 2As shown, it is a schematic flowchart of the server intelligent monitoring method for FTP files in this embodiment. The process includes at least the following steps: S1: Perform initial snapshot processing on several target folders in the FTP file server to obtain corresponding initial snapshot information and store it in the monitoring manager; S2: Connect the FTP file server to the client, and use the log system in the FTP file server to monitor the user's login event in real time. After detecting the user's login, obtain the file operation commands executed by the user through the log system; S3: Generate a monitoring process based on the execution of the file operation command in the target folder, and perform change snapshot processing on the target folder to obtain corresponding change snapshot information; S4: The monitoring process obtains the corresponding initial snapshot information from the monitoring manager and compares it with the change snapshot information to determine whether a file change has occurred in the corresponding target folder. After determining that a file change has occurred, import the changed file information into the data warehouse; S5: Statistically record the duration from when a single monitoring process sends a request to obtain the corresponding initial snapshot information to when it receives the initial snapshot information sent by the monitoring manager, and record it as the data transmission duration. Based on the data transmission durations corresponding to each monitoring process obtained within a preset operation cycle, determine whether the operation process of the monitoring manager is qualified; S6: When it is determined to be unqualified, determine and generate corresponding instructions based on each data transmission duration, determine to increase the operating memory of the monitoring manager based on the instructions, merge duplicate monitoring processes, update the initial snapshot information in the monitoring manager, or optimize the transmission environment.

[0027] Specifically, in this embodiment, the initial snapshot information includes the list of each file in the corresponding target folder and its metadata information, such as file name, size, modification time, etc.; the file operation commands include changing the directory, uploading files, deleting files, etc. for the files in the target folder. Among them, the commands related to the operations of each target folder are preferentially obtained and recorded. After obtaining the file operation commands through the logging system, it indicates that the corresponding target folder has performed the relevant operations of the file operation commands. At this time, a corresponding monitoring process can be generated according to a file operation command; the target folder is processed for a changed snapshot through the monitoring process. The changed snapshot information includes the list of each file and its metadata information after the file operation command is executed on the target folder. At this time, the file name, size, modification time, etc. will change accordingly; the storage method of the initial snapshot information in the monitoring manager is distributed storage, and each initial snapshot information does not affect each other and is uniformly allocated by the centralized monitoring manager; in this embodiment, the logging system in the FTP file server is used to monitor the user's login events in real time. In an alternative embodiment, the user's login events can also be monitored through the application programming interface (API). Through the snapshot mechanism and asynchronous notification, the present invention reduces unnecessary real-time queries and short-interval timed polling (such as monitoring every second), thereby greatly reducing the pressure on the FTP file server and improving the stability and availability of the server; in addition, the combination of the snapshot mechanism and the asynchronous notification technology realizes efficient and real-time monitoring of file changes on the FTP file server, breaking through the bottleneck of large server pressure caused by real-time queries in traditional methods and solving the problem of insufficient timeliness of data import caused by too long polling intervals. And the data transmission duration used by the monitoring process to obtain the initial snapshot information from the monitoring manager can be used to determine that the operation process of the monitoring manager is unqualified. Furthermore, by generating corresponding instructions to adjust the parameters in each module or sending corresponding notifications, the operation process of the monitoring manager is optimized, effectively shortening the data transmission duration, thereby improving the efficiency and accuracy of importing changed files from the FTP file server to the data warehouse.

[0028] Please refer to Figure 3 As shown, it is a logical decision diagram for determining whether the operation process of the monitoring manager for the duration mean is qualified and the corresponding processing in this embodiment. The process of determining whether the operation process of the monitoring manager is qualified based on the data transmission durations corresponding to each monitoring process obtained within the preset operation cycle includes: counting each data transmission duration and calculating the average value to obtain the duration mean; making a determination based on the comparison result between the duration mean and the preset duration mean, or re-determining whether the operation process of the monitoring manager is qualified based on the total number of monitoring processes generated within the preset operation cycle; when the determination is unqualified, determine the reason for the unqualified based on the difference between the duration mean and the preset duration mean.

[0029] Specifically, in this embodiment, by analyzing the historical data collected in the past, and combining statistical methods and application scenarios, the corresponding preset or critical parameter values are determined. The application scenario is the synchronization of large file metadata and the import of cross-regional files into the data warehouse. The preset duration average H0 can be divided into the first preset duration average H1 and the second preset duration average H2. Set H1 = 70ms and H2 = 80ms. The specific process of comparing the duration average H with H1 and H2 is as follows: When H is less than or equal to H1, it indicates that the current duration average is relatively small. Therefore, it can be determined that the data transfer durations of the vast majority within the preset operation cycle conform to the current preset standard. Therefore, it is determined that the operation process of the monitoring manager is qualified. When H is greater than H1 and less than or equal to H2, a new parameter - the total number of monitoring processes can be further introduced, and the operation process of the monitoring manager is re-determined based on the total number of monitoring processes generated within the preset operation cycle to improve the accuracy of the determination process. When H is greater than H2, it indicates that the current duration average is relatively large. At this time, it is determined that the data transfer durations of the vast majority within the preset cycle do not conform to the current preset standard. Therefore, it is directly determined that the operation process of the current monitoring manager is unqualified. When the determination is unqualified, the reason for the unqualified is determined according to the difference between the duration average H and the second preset duration average H2. At this time, H is not infinite. It can be understood that in the embodiments of the present invention, no specific limitations are made on H1 and H2. In an alternative embodiment, new H1 = 90ms and H2 = 100ms can also be set, as long as it is satisfied that the operation process of the monitoring manager is determined to be qualified or unqualified through the comparison results of H with H1 and H2. The specific comparison process refers to the above content.

[0030] Further, the process of re-determining whether the operation process of the monitoring manager is qualified based on the total number of the monitoring processes generated within the preset operation cycle includes: counting the total number of the monitoring processes and recording it as the total number of processes; determining whether to increase the operation memory of the monitoring manager based on the comparison result between the total number of processes and the critical total number of processes.

[0031] Specifically, in this embodiment, the total number of processes Q to be counted is carried out within a single preset operation cycle. The critical total number of processes Q0 is set to 80. The specific comparison process between Q and Q0 is as follows: When Q is less than or equal to Q0, it indicates that the total number of processes Q meets the current monitoring process quantity requirement. At this time, Q will not affect the data transmission duration. Therefore, if the situation where H is greater than H1 and less than or equal to H2 occurs, it is determined that the operation process of the monitoring manager is unqualified, and the reason needs to be determined based on the difference between the duration average value H and the second preset duration average value H2. When Q is greater than Q0, it indicates that the current Q does not meet the current monitoring process quantity requirement. At this time, Q has an impact on the data transmission duration during the operation process. Therefore, if the situation where H is greater than H1 and less than or equal to H2 occurs, the operation memory of the monitoring manager can be increased to eliminate the impact brought by excessive Q. At this time, Q is not infinitely large. It can be understood that in the embodiment of the present invention, no specific limitation is made on Q0. In an alternative embodiment, a new Q0 = 100 can also be set, as long as it meets the requirement of determining whether to increase the operation memory of the monitoring manager by comparing Q with Q0. The specific comparison process refers to the above content.

[0032] Further, the process of determining whether to increase the operation memory of the monitoring manager based on the comparison result between the total number of processes and the critical total number of processes includes: counting the total number of processes in each historical operation cycle, calculating the variance based on each total number of processes to obtain the process variance; determining to increase the operation memory based on the comparison result between the process variance and the preset process variance, and the increase amplitude of the operation memory has a negative correlation with the process variance.

[0033] Specifically, in this embodiment, if there is only an initial operation cycle, the total number of processes is the total number of processes Q in the above content. If there are multiple operation cycles, the total number of processes corresponding to each cycle is counted. At this time, the total number of processes in each case may not be the same in value. The process variance E is calculated based on each total number of processes. The preset process variance E0 can be divided into a first preset process variance E1 and a second preset process variance E2. Set E1 = 5.7 and E2 = 6.3. The specific comparison process between E and E1 and E2 is as follows: When E is less than or equal to E1 and greater than 4.5, it is determined to increase the original operating memory by 35%. When E is greater than 4.5 at this time, it is determined to increase the operating memory; when E is greater than E1 and less than or equal to E2, it is determined to increase the original operating memory by 20%; when E is greater than E2, it is determined to increase the original operating memory by 15%, and at this time E is not infinite; it should be noted that the smaller the value of E, the closer the number of monitoring processes in each historical operating cycle is. Therefore, if the total number of processes Q in the current preset operating cycle is relatively large, the total number of processes Q in each of the other operating cycles is also relatively large, indicating that the situation where the total number of processes Q exceeds the critical total number of processes Q0 occurs relatively frequently, and thus the increase range of the operating memory in the monitoring manager is also relatively large; it can be understood that in the embodiments of the present invention, E1 and E2 are not specifically limited. In an alternative embodiment, a new E1 = 6.2 and E2 = 6.8 can also be set, as long as it satisfies determining the increase range of the operating memory by comparing E with E1 and E2. The specific comparison process refers to the above content, and the increase range of the operating memory can also be set to other values that meet the requirements. For example, when E is less than or equal to E1, the increase range can also be set to 30%.

[0034] Please refer to Figure 4 As shown, it is a logical decision diagram for determining the reasons for the unqualified monitoring manager and the corresponding processing based on the time - length mean difference in this embodiment. The process of determining the reasons for unqualified according to the difference between the time - length mean and the preset time - length mean includes: calculating the difference between the time - length mean and the preset time - length mean and recording it as the time - length mean difference; determining the reasons for unqualified based on the comparison result between the time - length mean difference and the preset time - length mean difference; generating corresponding instructions based on the reasons, and based on the instructions, determining to merge the repeated monitoring processes, determining to update the initial snapshot information in the monitoring manager, or determining to optimize the transmission environment.

[0035] Specifically, in this embodiment, the preset time - length mean difference F0 can be divided into a first preset time - length mean difference F1 and a second preset time - length mean difference F2. Set F1 = 15ms and F2 = 30ms. The comparison process based on the time - length mean difference F with F1 and F2 is as follows: When F is less than or equal to F1, it indicates that F at this time is relatively small, that is, the difference between the average duration H and the second preset average duration H2 is relatively small. Therefore, it can be determined that the reason for non - compliance is that duplicate monitoring processes occur, resulting in the scheduling of a single initial snapshot information simultaneously, increasing the data transmission duration. At this time, the duplicate monitoring processes can be merged to reduce the number of acquisition requests for a single initial snapshot information. When F is greater than F1 and less than F2, it can be determined that the reason for non - compliance is that there is a problem with the transmission environment, resulting in an increase in the data transmission duration. At this time, the transmission environment can be optimized by increasing the compression ratio of the initial snapshot information. When F is greater than F2, and F at this time is not infinite, it indicates that F is relatively large. It can be determined that the reason for non - compliance is that the corresponding initial snapshot information is missing in the monitoring manager, resulting in the request for obtaining the initial snapshot information sent by the monitoring process not being responded to, increasing the data transmission duration. At this time, several target folders in the FTP file server can be traversed to re - obtain new initial snapshot information and replace and store it in the monitoring manager.

[0036] Further, the process of determining the merging process of the duplicate monitoring processes includes: counting the total number of the initial snapshot information in the monitoring manager within the preset operation cycle and recording it as the total snapshot information number; determining the merging standard based on the comparison result between the total snapshot information number and the preset total snapshot information number, and the merging standard has a negative correlation with the total snapshot information number; wherein, when the number of the duplicate monitoring processes is greater than or equal to the merging standard, the monitoring processes are merged.

[0037] Specifically, in this embodiment, the merging standard is the preset value of the duplicate monitoring processes. When the total snapshot information number T is less, for multiple monitoring processes, the probability of any two monitoring processes being duplicate is greater, and at the same time, it also indicates that the number of duplicate monitoring processes is more. Therefore, the merging standard that needs to be set should also be larger, so as to reduce the number of duplicate monitoring processes to the greatest extent, thereby ensuring that a monitoring process initiates a request to obtain the initial snapshot information to shorten the data transmission duration. The preset total snapshot information number T0 can be divided into the first preset total snapshot information number T1 and the second preset total snapshot information number T2. Set T1 = 900 and T2 = 1200. The specific comparison process based on T, T1, and T2 is as follows: When T is less than or equal to T1, it is determined that the set merging criterion is 50. It is clear that the minimum value of T at this time should satisfy the determination to enable the merging criterion. When T is greater than T1 and less than or equal to T2, it is determined that the set merging criterion is 30. When T is greater than T2, it is determined that the set merging criterion is 20, that is, when the number of repeated monitoring processes exceeds 20, they are merged, and T at this time is not infinite. It can be understood that in the embodiment of the present invention, T1 and T2 are not specifically limited. In an alternative embodiment, new values of T1 = 950 and T2 = 1250 can also be set, as long as the merging criterion can be determined by comparing T with T1 and T2. The specific comparison process refers to the above content. At the same time, the merging criterion can also be set to other values that meet the requirements. For example, when T is less than or equal to T1, it can also be set to 45.

[0038] Further, the process of determining to optimize the transmission environment includes: counting the number of the initial snapshot information simultaneously sent by the monitoring manager per unit time and recording it as the snapshot concurrency; determining to increase the compression rate of the initial snapshot information based on the comparison result between the snapshot concurrency and the preset snapshot concurrency, and the increase range of the compression rate is positively correlated with the snapshot concurrency.

[0039] Specifically, in this embodiment, the compression rate for the initial snapshot information is determined by the snapshot concurrency D, thereby reducing the data volume and bandwidth occupancy, and thus optimizing the transmission environment to reduce the data transmission duration. The preset snapshot concurrency D0 can be divided into a first preset snapshot concurrency D1 and a second preset snapshot concurrency D2. Set D1 = 40 and D2 = 55. The specific comparison process based on D with D1 and D2 is as follows: If D is less than or equal to D1 and greater than 20, it is determined to increase by 10% on the basis of the original compression rate. When D is greater than 20 at this time, it is determined to increase the compression rate. If D is greater than D1 and less than or equal to D2, it is increased by 15% on the basis of the original compression rate. If D is greater than D2, it is increased by 20% on the basis of the original compression rate. At this time, D is not infinite. It is clear that the increase range of the compression rate is not infinite and needs to satisfy ensuring the integrity of the initial snapshot information. The increase range can also be set to other values that meet the requirements. For example, when D is greater than D2, the increase range is 18%. It can be understood that in the embodiment of the present invention, D1 and D2 are not specifically limited. In an alternative embodiment, new values of D1 = 35 and D2 = 50 can also be set, as long as the increase range of the compression rate can be determined by comparing D with D1 and D2. The specific comparison process refers to the above content.

[0040] Further, after comparing the initial snapshot information with the changed snapshot information, the monitoring process stores the changed snapshot information in the monitoring manager and replaces it with new initial snapshot information as the object for the next comparison. In this way, the snapshot information in the monitoring manager can be continuously updated to ensure that the latest snapshot information is used as the comparison object in each comparison process.

[0041] Further, when the logging system captures that multiple file operation commands operate on each target folder, multiple corresponding monitoring processes are generated. Each monitoring process is connected to the monitoring manager; the paths where the target folders for operation are located are obtained and sent to the monitoring manager, and the monitoring manager sends the initial snapshot information corresponding to each path to the corresponding monitoring process. Through a centralized monitoring manager, unified management and scheduling of monitoring tasks are achieved, improving the flexibility and scalability of the system.

[0042] Further, the process of importing the changed file information into the data warehouse includes: the monitoring manager triggers an asynchronous notification mechanism and notifies the monitoring process, and the monitoring process imports the changed file information into the data warehouse; wherein, the asynchronous notification mechanism is a preset notification rule, including a message queue or a network hook. When a file change is detected, the asynchronous notification mechanism is triggered, and according to the preset notification rule, the changed file information is asynchronously sent to the running monitoring process, and then the changed file information is imported into the data warehouse or the data lake through the monitoring process.

[0043] It can be understood that in the embodiments of the present invention, no specific limitations are imposed on any preset parameter or critical parameter, and the above-mentioned values are not limited thereto. Those skilled in the art can adjust the preset parameters or critical parameters according to actual needs, or the analysis of historical data, or the usage of the device.

[0044] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0045] The above are only the preferred embodiments of the present invention and are not used to limit the present invention; for those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent monitoring method for FTP files on a server, characterized in that, Including: Performing initial snapshot processing on several target folders in the FTP file server to obtain corresponding initial snapshot information and storing it in the monitoring manager; Connecting the FTP file server to the client, monitoring the user's login events in real time through the log system in the FTP file server, and after detecting the user's login, obtaining the file operation commands executed by the user through the log system; Generating a monitoring process based on the execution of the file operation command in the target folder, and performing change snapshot processing on the target folder to obtain corresponding change snapshot information; The monitoring process obtains the corresponding initial snapshot information from the monitoring manager and compares it with the change snapshot information to determine whether a file change has occurred in the corresponding target folder. After determining that a file change has occurred, the changed file information is imported into the data warehouse; Counting the duration from when a single monitoring process sends a request to obtain the corresponding initial snapshot information until it receives the initial snapshot information sent by the monitoring manager and recording it as the data transmission duration, and determining whether the operation process of the monitoring manager is qualified based on the data transmission durations corresponding to each monitoring process obtained within a preset operation cycle; When determining that it is unqualified, generating corresponding instructions based on each data transmission duration, determining to increase the operating memory of the monitoring manager based on the instructions, merging duplicate monitoring processes, updating the initial snapshot information in the monitoring manager, or optimizing the transmission environment.

2. The server intelligent monitoring method for FTP files according to claim 1, characterized in that, The process of determining whether the operation process of the monitoring manager is qualified based on the data transmission durations corresponding to each monitoring process obtained within the preset operation cycle includes: Counting each data transmission duration and calculating the average value to obtain the duration average value; Making a determination based on the comparison result between the duration average value and the preset duration average value, or re-determining whether the operation process of the monitoring manager is qualified based on the total number of monitoring processes generated within the preset operation cycle; When determining that it is unqualified, determining the cause of unqualified according to the difference between the duration average value and the preset duration average value.

3. The server intelligent monitoring method for FTP files according to claim 2, characterized in that, The process of re-determining whether the operation process of the monitoring manager is qualified based on the total number of monitoring processes generated within the preset operation cycle includes: Counting the total number of monitoring processes and recording it as the total process number; Determining whether to increase the operating memory of the monitoring manager based on the comparison result between the total process number and the critical total process number.

4. The server intelligent monitoring method for FTP files according to claim 3, characterized in that, The process of determining whether to increase the operating memory of the monitoring manager based on the comparison result between the total process number and the critical total process number includes: Counting the total number of processes in each historical operation cycle, calculating the variance based on each total process number to obtain the process variance; Determining to increase the operating memory based on the comparison result between the process variance and the preset process variance, and the increase amplitude of the operating memory is negatively correlated with the process variance.

5. The server intelligent monitoring method for FTP files according to claim 2, wherein The process of determining the cause of unqualified according to the difference between the duration average value and the preset duration average value includes: Calculating the difference between the duration average value and the preset duration average value and recording it as the duration average difference; Determine the reason for non - compliance based on the comparison result between the average duration difference and the preset average duration difference; Generate corresponding instructions based on the reason, and based on the instructions, determine to merge the repeated monitoring processes, determine to update the initial snapshot information in the monitoring manager, or determine to optimize the transmission environment.

6. The server intelligent monitoring method for FTP files according to claim 5, characterized in that, The process of determining to merge the repeated monitoring processes includes: Count the total number of the initial snapshot information in the monitoring manager within the preset operation cycle and record it as the total number of snapshot information; Determine the merging criterion based on the comparison result between the total number of snapshot information and the preset total number of snapshot information. The merging criterion has a negative correlation with the total number of snapshot information; Among them, when the number of repeated monitoring processes is greater than or equal to the merging criterion, the monitoring processes are merged.

7. The server intelligent monitoring method for FTP files according to claim 6, characterized in that The process of determining to optimize the transmission environment includes: Count the number of the initial snapshot information sent simultaneously by the monitoring manager per unit time and record it as the snapshot concurrency; Determine to increase the compression ratio of the initial snapshot information based on the comparison result between the snapshot concurrency and the preset snapshot concurrency. The increase amplitude of the compression ratio has a positive correlation with the snapshot concurrency.

8. The server intelligent monitoring method for FTP files according to claim 1, characterized in that, After completing the comparison between the initial snapshot information and the changed snapshot information, the monitoring process stores the changed snapshot information in the monitoring manager and replaces it with new initial snapshot information as the object for the next comparison.

9. The server intelligent monitoring method for FTP files according to claim 1, characterized in that When the log system captures that multiple file operation commands operate on each target folder, multiple corresponding monitoring processes are generated, and each monitoring process is connected to the monitoring manager; Obtain the paths where each target folder for operation is located, and send each path to the monitoring manager. The monitoring manager sends the initial snapshot information corresponding to each path to the corresponding monitoring process respectively.

10. The server intelligent monitoring method for FTP files according to claim 1, characterized in that, The process of importing the changed file information into the data warehouse includes: The monitoring manager triggers an asynchronous notification mechanism and notifies the monitoring process, and the monitoring process imports the changed file information into the data warehouse; Among them, the asynchronous notification mechanism is a preset notification rule, including a message queue or a network hook.

Images