A server intelligent monitoring method for FTP files

Abstract

The application relates to the technical field of FTP file management, in particular to a server intelligent monitoring method for FTP files. The method carries out snapshot processing on a target folder in an FTP file server and stores initial snapshot information to a monitoring manager; after a file operation command on the target folder is monitored, a monitoring process is generated and the target folder is processed to obtain changed snapshot information; the initial snapshot information and the changed snapshot information are compared to determine whether the files in the target folder have changed; after the change, the file information is imported into a data warehouse through the monitoring process, thereby reducing the pressure of the FTP file server; the data transmission time length of the initial snapshot information obtained by the monitoring process is counted, the time length average is obtained based on the data transmission time lengths, and the running process of the monitoring manager is determined; when unqualified, corresponding instructions are generated and corresponding parameters are determined, the data transmission time length is shortened, and the efficiency of importing the changed files into the data warehouse is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of FTP file management, and particularly relates to a server intelligent monitoring method for FTP files. BACKGROUND

[0002] In big data processing, as a traditional file transmission method, the file transfer protocol (FTP) still occupies an important position in the construction of a data warehouse. In the data import process, specific data files often need to be read from an FTP file server. However, due to network delay, file generation delay, transmission error, or other unpredictable reasons, the target file to be read may not exist or be ready, which brings great challenges to the data import process. At present, for the monitoring of FTP file server file changes, the methods of timing polling or real-time monitoring are usually used. Timing polling is to constantly query the file list on the FTP server to determine whether the target file exists or has changed by setting a fixed time interval. However, this method has obvious defects: on the one hand, if the polling interval is too short (such as listening every second), it will cause excessive pressure on the FTP server, leading to a decline in server performance, and even affecting other normal file transmission operations; on the other hand, if the polling interval is too long, it may lead to insufficient timeliness of data import, which cannot meet the real-time requirements of business data. The real-time monitoring technology can solve the timeliness problem to some extent, but it also faces the problem of resource waste.

[0003] The prior art Chinese patent No. CN1997019B provides a message monitoring receiving method based on FTP transmission. The technical solution sends a file to an FTP server through an FTP client, and analyzes and judges the information and content of the file in real time to determine whether it meets the predetermined standard. The file meeting the predetermined standard is sent to a coding and decoding module, the file sent by the FTP server is decoded and analyzed by the coding and decoding module to obtain the information contained in the file, and the display storage module displays the file information in real time according to the analysis result of the coding and decoding module and stores the file into a database, so as to timely understand the situation of the monitored user, and improve the speed and efficiency of subsequent analysis, query and processing of the message by storing the message into the database in a formatted manner. The technical solution also determines the subsequent operation by real-time analysis of the file, which causes excessive running pressure of the FTP server, and does not involve how to optimize the data transmission process based on the data transmission time length, which causes low data transmission efficiency. SUMMARY

[0004] To this end, the application provides a server intelligent monitoring method for FTP files to solve the problem that the prior art has not considered the data transmission duration based on snapshot information to determine the running process of the monitoring manager and make corresponding parameter adjustment based on the determination result, thereby resulting in low efficiency of importing file information into the data warehouse.

[0005] To achieve the above-mentioned object, the application provides a server intelligent monitoring method for FTP files, comprising:

[0006] Respectively performing initial snapshot processing on a plurality of target folders in the FTP file server to obtain a plurality of corresponding initial snapshot information and store it into the monitoring manager;

[0007] Connecting the FTP file server and the user end, monitoring the user's login event in real time through the log system in the FTP file server, and after monitoring the user login, obtaining the file operation command executed by the user through the log system;

[0008] Based on the monitoring process generated after executing the file operation command in the target folder, performing change snapshot processing on the target folder to obtain the corresponding change snapshot information;

[0009] The monitoring process obtains the corresponding initial snapshot information from the monitoring manager and compares it with the change snapshot information to determine whether file changes have occurred in the corresponding target folder, and after determining that file changes have occurred, imports the change file information into the data warehouse;

[0010] Statistically determining the duration from the time when the monitoring process sends a request for the corresponding initial snapshot information to the time when the monitoring manager sends the initial snapshot information, and recording it as the data transmission duration, and determining whether the running process of the monitoring manager is qualified based on the data transmission duration corresponding to each monitoring process obtained within the preset running period;

[0011] When it is determined to be unqualified, based on each data transmission duration, determine to generate a corresponding instruction, based on the instruction, determine to increase the running memory of the monitoring manager, combine the repeated monitoring processes, update the initial snapshot information in the monitoring manager, or optimize the transmission environment.

[0012] Further, the process of determining whether the running process of the monitoring manager is qualified based on each data transmission duration corresponding to each monitoring process obtained within the preset running period comprises:

[0013] Statistically determining each data transmission duration and performing average value calculation to obtain the duration average value;

[0014] determining whether the running process of the monitoring manager is qualified based on a comparison result of the time length average and a preset time length average, or re-determining whether the running process of the monitoring manager is qualified based on a total number of the monitoring processes generated in a preset running period;

[0015] when determining that the running process is unqualified, determining an unqualified reason according to a difference between the time length average and the preset time length average.

[0016] Further, the process of re-determining whether the running process of the monitoring manager is qualified based on the total number of the monitoring processes generated in the preset running period comprises:

[0017] counting the total number of the monitoring processes and recording it as a process total number;

[0018] determining whether to increase the running memory of the monitoring manager based on a comparison result of the process total number and a critical process total number.

[0019] Further, the process of determining whether to increase the running memory of the monitoring manager based on the comparison result of the process total number and the critical process total number comprises:

[0020] counting the process total number in each historical running period, and performing variance calculation based on each process total number to obtain a process variance;

[0021] determining to increase the running memory based on a comparison result of the process variance and a preset process variance, and a negative correlation relationship between an increase amplitude of the running memory and the process variance.

[0022] Further, the process of determining the unqualified reason according to the difference between the time length average and the preset time length average comprises:

[0023] calculating the difference between the time length average and the preset time length average and recording it as a time length average difference;

[0024] determining the unqualified reason based on a comparison result of the time length average difference and a preset time length average difference;

[0025] generating a corresponding instruction based on the reason, determining to perform a merging process on the repeated monitoring processes, determining to update the initial snapshot information in the monitoring manager, or determining to optimize a transmission environment based on the instruction.

[0026] Further, the process of determining to perform the merging process on the repeated monitoring processes comprises:

[0027] counting a total number of the initial snapshot information in the monitoring manager in the preset running period and recording it as a snapshot information total number;

[0028] Determine a merging standard based on the comparison result of the total number of snapshot information and a preset total number of snapshot information, the merging standard being negatively correlated with the total number of snapshot information.

[0029] When the number of repeated monitoring processes is greater than or equal to the merging standard, the monitoring processes are merged.

[0030] Further, the process of optimizing the transmission environment includes:

[0031] Count the number of initial snapshot information sent simultaneously in the monitoring manager in a unit of time and record it as snapshot concurrency;

[0032] Determine to increase the compression rate of the initial snapshot information based on the comparison result of the snapshot concurrency and a preset snapshot concurrency, the increase amplitude of the compression rate being positively correlated with the snapshot concurrency.

[0033] Further, after the comparison of the initial snapshot information and the changed snapshot information is completed, the monitoring process stores the changed snapshot information into the monitoring manager and replaces it with new initial snapshot information as the object of next comparison.

[0034] Further, when the log system captures that multiple file operation commands operate on each target folder, multiple monitoring processes are generated correspondingly, each monitoring process being connected with the monitoring manager.

[0035] Obtain the paths of each target folder being operated and send each path to the monitoring manager, the monitoring manager sending the initial snapshot information corresponding to each path to the corresponding monitoring process.

[0036] Further, the process of importing the changed file information into the data warehouse includes:

[0037] The monitoring manager triggers an asynchronous notification mechanism and notifies the monitoring process, and the monitoring process imports the changed file information into the data warehouse.

[0038] The asynchronous notification mechanism is a preset notification rule, including a message queue or a network hook.

[0039] Compared with the prior art, the server intelligent monitoring method for FTP files has the beneficial effects that: the method obtains initial snapshot information by snapshot processing on a plurality of target folders in the FTP file server and stores the initial snapshot information into a monitoring manager, generates a monitoring process and obtains change snapshot information by snapshot processing on the target folder after monitoring that a user has executed a file operation command on the target folder, compares the initial snapshot information with the change snapshot information to determine whether a file in the corresponding target folder has changed, and imports the change file information into a data warehouse through the monitoring process after determining that the change has occurred, the method monitors the user's file operation command to determine whether to import the change file information into the data warehouse, thereby reducing the pressure on the FTP file server; at the same time, the time length used for the monitoring process to send a request to obtain the corresponding initial snapshot information to the monitoring manager and send the initial snapshot information is recorded as a data transmission time length, the time length average is obtained through a plurality of data transmission time lengths in a preset running period, and whether the running process of the monitoring manager is qualified is determined based on the time length average, when it is determined that the running process is not qualified, corresponding instructions can be generated and the corresponding parameters can be determined based on the instructions, thereby shortening each data transmission time length, and the efficiency and accuracy of importing the files in the FTP file server into the data warehouse can be improved.

[0040] Further, based on the comparison result of the time length average and the preset time length average to determine whether the running process of the monitoring manager is qualified, the total number of monitoring processes generated in the preset running period is further obtained, and the process total number is obtained, and the running process of the monitoring manager is determined again based on the comparison result of the process total number and the critical process total number, thereby improving the accuracy of the determination process.

[0041] Further, when it is determined to increase the running memory in the monitoring manager based on the comparison result of the process total number and the critical process total number, the process variance is calculated by obtaining the process total number in a plurality of historical running periods, the increase range of the running memory is determined based on the comparison of the process variance and the preset process variance, and each data transmission time length can be shortened, thereby improving the efficiency of importing the change file information from the FTP file server into the data warehouse.

[0042] Further, the application also acquires the time length mean difference, and determines the unqualified reason of the running process of the monitoring manager based on the comparison result of the time length mean difference and the preset time length mean difference, and then can perform corresponding processing, when it is determined that repeated monitoring processes appear, it needs to be combined, when it is determined that the transmission environment has problems, it needs to increase the compression rate of the initial snapshot information, when it is determined that the corresponding initial snapshot information is missing in the monitoring manager, a plurality of target folders in the FTP file server are traversed to reacquire new initial snapshot information and replace the storage in the monitoring manager, and then the data transmission time is shortened, thereby improving the import efficiency of the change file information.

[0043] Further, the application also determines the combination standard of combining the repeated monitoring processes based on the comparison result of the total number of snapshot information and the preset total number of snapshot information, and then the data transmission time is shortened, thereby improving the import efficiency of the change file information.

[0044] Further, the application also determines the amplitude of increasing the compression rate of the initial snapshot information based on the comparison result of the snapshot concurrency and the preset snapshot concurrency, and then the data transmission time is shortened, thereby improving the import efficiency of the change file information. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 It is a module schematic diagram of the intelligent monitoring system for the FTP file server in the application;

[0046] Figure 2 It is a flow schematic diagram of the intelligent monitoring method for the FTP file in the application;

[0047] Figure 3 It is a logic determination diagram for determining whether the running process of the monitoring manager is qualified based on the time length mean value and corresponding processing in the application;

[0048] Figure 4 It is a logic determination diagram for determining the unqualified reason of the monitoring manager based on the time length mean difference and corresponding processing in the application. DETAILED DESCRIPTION

[0049] In order to make the purpose and advantages of the application more clear and obvious, the application will be further described below in combination with examples; it should be understood that the specific examples described here are only used to explain the application, and not to limit the protection scope of the application.

[0050] The preferred embodiments of the application will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the application, and are not intended to limit the protection scope of the application.

[0051] It should be noted that in the description of the present application, unless otherwise explicitly specified and limited, the term "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be internal communication of two elements. For those skilled in the art, the specific meaning of the above-mentioned terms in the present application can be understood according to the specific circumstances.

[0052] Please refer to Figure 1As shown, it is a module schematic diagram of the server intelligent monitoring system for FTP files in the embodiment. It comprises: an FTP file server, a monitoring manager, an initial snapshot module, a monitoring process module, a data import module, a statistics module, an analysis module, an instruction generation module, and an adjustment module.The initial snapshot module is connected with the FTP file server and the monitoring manager respectively, to perform initial snapshot processing on a plurality of target folders in the FTP file server respectively to obtain a plurality of initial snapshot information respectively, and store the plurality of initial snapshot information in the monitoring manager, wherein the initial snapshot processing is the first static recording of the target folder; a log system in the FTP file server is used to monitor the login event of a user in real time, and after monitoring the login of the user, the log system is used to capture and record the file operation command executed by the user; the monitoring process module is connected with the FTP file server and the monitoring manager respectively, to generate a corresponding monitoring process in the case that a file operation command is determined to be executed on a plurality of target folders, each monitoring process is used to perform change snapshot processing on the corresponding target folder to obtain corresponding change snapshot information, and is used to obtain the initial snapshot information of the corresponding target folder from the monitoring manager, and compare the initial snapshot information with the change snapshot information, wherein the change snapshot processing is the recording of the modification or update of the target file which executes the file operation command; the data import module is connected with the monitoring process module, to import the change file information into the data warehouse in the case that the file in the corresponding target folder is determined to be changed; the statistical module is connected with the monitoring process module and the monitoring manager respectively, to count the data transmission time corresponding to each monitoring process module obtained within a preset running period, wherein the data transmission time is the time length from the time when a single monitoring process module sends a request for obtaining the corresponding initial snapshot information to the time when the initial snapshot information sent by the monitoring manager is received; the analysis module is connected with the statistical module, to determine whether the running process of the monitoring manager is qualified based on the obtained data transmission time, and determine the reason in the case that the running process is not qualified, wherein the reason includes that a repeated monitoring process occurs to cause a single initial snapshot information to be dispatched simultaneously to increase the data transmission time, or the number of monitoring processes is too large to cause the running memory in the monitoring manager to be insufficient to increase the data transmission time, or the corresponding initial snapshot information is missing in the monitoring manager to cause the request for obtaining the initial snapshot information sent by the monitoring process to be unable to be responded to increase the data transmission time, or the transmission environment has a problem to increase the data transmission time; the instruction generation module is connected with the analysis module, to generate a corresponding instruction based on the reason; the adjustment module is connected with the instruction generation module, the monitoring manager, and the monitoring process module, to determine to increase the running memory of the monitoring manager, to determine to merge the repeated monitoring processes, to determine to update the initial snapshot information in the monitoring manager, or to determine to optimize the transmission environment based on the instruction.

[0053] Please refer to Figure 2As shown, it is a flow diagram of the server intelligent monitoring method for FTP files in the present embodiment. The flow includes at least the following steps:

[0054] S1: Perform initial snapshot processing on each target folder in the FTP file server to obtain corresponding initial snapshot information and store it in the monitoring manager;

[0055] S2: Connect the FTP file server with the user terminal, monitor the user's login event in real time through the log system in the FTP file server, and after monitoring the user login, obtain the file operation command executed by the user through the log system;

[0056] S3: Based on the file operation command executed in the target folder, generate a monitoring process and perform change snapshot processing on the target folder to obtain corresponding change snapshot information;

[0057] S4: The monitoring process obtains the corresponding initial snapshot information from the monitoring manager and compares it with the change snapshot information to determine whether file changes have occurred in the corresponding target folder, and after determining that file changes have occurred, import the changed file information into the data warehouse;

[0058] S5: Statistic the duration from the request for the corresponding initial snapshot information to the receipt of the initial snapshot information sent by the monitoring manager, and record it as the data transmission duration. Based on the data transmission duration corresponding to each monitoring process obtained within the preset running period, determine whether the running process of the monitoring manager is qualified;

[0059] S6: When it is determined to be unqualified, determine the corresponding instructions based on each data transmission duration, increase the running memory of the monitoring manager based on the instructions, merge the repeated monitoring processes, update the initial snapshot information in the monitoring manager, or optimize the transmission environment.

[0060] Specifically, in the embodiment, the initial snapshot information includes a list of files in the target folder and metadata information thereof, such as file name, size, modification time, etc.; the file operation command includes changing the directory of the file in the target folder, uploading the file, deleting the file, etc., wherein the command related to the operation of each target folder is preferentially acquired and recorded, and after the file operation command is acquired through the log system, it is indicated that the corresponding target folder has performed the related operation of the file operation command, at this time, one corresponding monitoring process can be generated according to one file operation command; the target folder is processed by the monitoring process to change the snapshot, the changed snapshot information includes a list of files in the target folder after the file operation command is executed, and the file name, size, modification time, etc. at this time will change accordingly; the storage mode of the initial snapshot information in the monitoring manager is distributed storage, and each initial snapshot information complements each other and is uniformly deployed by the centralized monitoring manager; in the embodiment, the login event of the user is monitored in real time through the log system in the FTP file server, and in an optional embodiment, the login event of the user can also be monitored through an application programming interface (API). The present application reduces unnecessary real-time queries and unnecessary short-interval timing polling (such as every second monitoring) through the snapshot mechanism and asynchronous notification, thereby greatly reducing the pressure on the FTP file server and improving the stability and availability of the server; in addition, the snapshot mechanism and asynchronous notification technology are combined to realize efficient and real-time monitoring of the changes of the FTP file server, breaking through the bottleneck of high server pressure caused by real-time queries in traditional methods and solving the problem of insufficient timeliness of data import caused by too long polling interval. The data transmission time length used by the monitoring process to acquire the initial snapshot information from the monitoring manager can determine that the running process of the monitoring manager is unqualified, and then corresponding instructions are generated to adjust the parameters in each module or issue corresponding notifications, so as to optimize the running process of the monitoring manager, effectively shorten the data transmission time length, and thereby improve the efficiency and accuracy of importing changed files from the FTP file server to the data warehouse.

[0061] Please refer to Figure 3 The logic determination diagram for determining whether the running process of the monitoring manager is qualified and the corresponding processing in the embodiment is shown in FIG. 6. The process of determining whether the running process of the monitoring manager is qualified based on the data transmission time length corresponding to each monitoring process acquired within the preset running period includes: calculating the average value of the data transmission time length to obtain the average time length; determining based on the comparison result of the average time length and the preset average time length, or re-determining whether the running process of the monitoring manager is qualified based on the total number of monitoring processes generated within the preset running period; when it is determined to be unqualified, determining the unqualified reason according to the difference between the average time length and the preset average time length.

[0062] Specifically, in the embodiment, the corresponding preset or critical parameter value is determined by analyzing the historical data collected in the past, combining statistical methods, and application scenarios. The application scenario is large file metadata synchronization and cross-regional file import into the data warehouse. The preset time length average H0 can be divided into a first preset time length average H1 and a second preset time length average H2. H1 = 70 ms and H2 = 80 ms are set. The comparison process based on the time length average H and H1 and H2 is as follows:

[0063] When H is less than or equal to H1, it indicates that the current time length average is relatively small, so it can be determined that the majority of data transmission time length in the preset running period meets the current preset standard, and therefore, it is determined that the running process of the monitoring manager is qualified. When H is greater than H1 and less than or equal to H2, a new parameter, the total number of monitoring processes, can be further introduced to re-determine the running process of the monitoring manager based on the total number of monitoring processes generated in the preset running period to improve the accuracy of the determination process. When H is greater than H2, it indicates that the current time length average is relatively large, so it is determined that the majority of data transmission time length in the preset period does not meet the current preset standard, and therefore, it is directly determined that the running process of the current monitoring manager is unqualified. When it is determined to be unqualified, the reason for unqualification is determined according to the difference between the time length average H and the second preset time length average H2. At this time, H is not infinite. It can be understood that H1 and H2 are not specifically limited in the embodiment. In an optional embodiment, a new H1 = 90 ms and H2 = 100 ms can be set. Only the comparison result of H and H1 and H2 is required to determine whether the running process of the monitoring manager is qualified. The specific comparison process is described above.

[0064] Further, the process of re-determining whether the running process of the monitoring manager is qualified based on the total number of monitoring processes generated in the preset running period includes: counting the total number of monitoring processes and recording it as the total number of processes; and determining whether to increase the running memory of the monitoring manager based on the comparison result of the total number of processes and the critical total number of processes.

[0065] Specifically, in the embodiment, the total number of processes Q is counted in a single preset running period, the critical total number of processes Q0 is set to 80, and the comparison process based on Q and Q0 is specifically as follows: when Q is less than or equal to Q0, it is indicated that the total number of processes Q meets the current requirement of the number of monitoring processes, and Q at this time does not affect the data transmission time length, and therefore, if H is greater than H1 and less than or equal to H2, it is determined that the running process of the monitoring manager is unqualified, and the reason needs to be determined based on the difference between the time length average H and the second preset time length average H2. When Q is greater than Q0, it is indicated that the current Q does not meet the current requirement of the number of monitoring processes, and Q at this time has an impact on the data transmission time length in the running process, and therefore, if H is greater than H1 and less than or equal to H2, the influence of excessive Q can be eliminated by increasing the running memory of the monitoring manager, and Q is not infinite at this time. It can be understood that Q0 is not specifically limited in the embodiment, and in an optional embodiment, a new Q0=100 can be set, as long as the comparison between Q and Q0 is used to determine whether the running memory of the monitoring manager is increased, and the specific comparison process is referred to the above content.

[0066] Further, the process of determining whether to increase the running memory of the monitoring manager based on the comparison result of the total number of processes and the critical total number of processes includes: counting the total number of processes in each historical running period, calculating the process variance based on each total number of processes to obtain a process variance; and determining to increase the running memory based on the comparison result of the process variance and a preset process variance, and the increase amplitude of the running memory is in a negative correlation with the process variance.

[0067] Specifically, in the embodiment, if there is only an initial running period, the total number of processes is the total number of processes Q in the above content, and if there are multiple running periods, the corresponding total number of processes in each period is counted, and at this time, the total number of processes may not be the same in value, the process variance E is calculated based on each total number of processes, the preset process variance E0 can be divided into a first preset process variance E1 and a second preset process variance E2, E1=5.7 and E2=6.3 are set, and the comparison process based on E, E1 and E2 is specifically as follows:

[0068] When E is less than or equal to E1 and greater than 4.5, it is determined to increase the running memory by 35% on the basis of the original running memory, and when E is greater than 4.5, it is determined to increase the running memory; when E is greater than E1 and less than or equal to E2, it is determined to increase the running memory by 20% on the basis of the original running memory; when E is greater than E2, it is determined to increase the running memory by 15% on the basis of the original running memory, and E is not infinite at this time; it should be noted that the smaller the value of E is, the closer the number of listening processes in each historical running period is, and therefore, if the total number of processes Q in the current preset running period is relatively large, the total number of processes Q in each of the remaining running periods is also relatively large, indicating that the case that the total number of processes Q exceeds the critical total number of processes Q0 occurs relatively frequently, and the increase range of the running memory in the listening manager is also relatively large; it can be understood that E1 and E2 are not specifically limited in the embodiment of the application, and in an optional embodiment, a new E1=6.2 and E2=6.8 can also be set, as long as the increase range of the running memory is determined by comparing E with E1 and E2, the specific comparison process is referred to the above content, and the increase range of the running memory can also be set to other required values, for example, the increase range can also be set to 30% in the case that E is less than or equal to E1.

[0069] Please refer to Figure 4 is shown, which is a logic determination diagram of reasons for determining that the listening manager is unqualified and corresponding processing based on the time length average difference in the embodiment. The process of determining the reason for unqualification according to the difference between the time length average and the preset time length average includes: calculating the difference between the time length average and the preset time length average and recording it as a time length average difference; determining the reason for unqualification based on the comparison result of the time length average difference and the preset time length average difference; generating a corresponding instruction based on the reason, determining to perform a merging process on the repeated listening processes, determining to update the initial snapshot information in the listening manager, or determining to optimize the transmission environment.

[0070] Specifically, in the embodiment, the preset time length average difference F0 can be divided into a first preset time length average difference F1 and a second preset time length average difference F2, F1=15 ms and F2=30 ms are set, and the comparison process of the time length average difference F with F1 and F2 is specifically as follows:

[0071] When F is less than or equal to F1, it indicates that F at this time is relatively small, that is, the difference between the time average H and the second preset time average H2 is relatively small, and thus it is determined that the reason for unqualification is that repeated monitoring processes occur to cause a single initial snapshot information to be simultaneously scheduled to increase the data transmission time, and thus the repeated monitoring processes can be combined to reduce the number of acquisition requests for the single initial snapshot information. When F is greater than F1 and less than F2, it is determined that the reason for unqualification is that the transmission environment is problematic to increase the data transmission time, and thus the transmission environment can be optimized by increasing the compression rate of the initial snapshot information. When F is greater than F2, F at this time is not infinite, which indicates that F at this time is relatively large, and it is determined that the reason for unqualification is that the corresponding initial snapshot information is missing in the monitoring manager to cause the monitoring process to send a request for acquiring the initial snapshot information to be unable to be responded to, thereby increasing the data transmission time, and thus the target folders in the FTP file server can be traversed to reacquire new initial snapshot information and replace the initial snapshot information stored in the monitoring manager.

[0072] Further, the process of determining the repeated monitoring processes to be combined includes: counting the total number of the initial snapshot information in the monitoring manager in the preset running period and recording it as a snapshot information total number; determining a combination standard based on the comparison result of the snapshot information total number and a preset snapshot information total number, the combination standard being in a negative correlation with the snapshot information total number; and when the number of the repeated monitoring processes is greater than or equal to the combination standard, the monitoring processes are combined.

[0073] Specifically, in the embodiment, the combination standard is a preset number of repeated monitoring processes. When the snapshot information total number T is less, the probability of any two monitoring processes being repeated is greater when facing multiple monitoring processes, and it also indicates that the number of repeated monitoring processes is greater. Therefore, the combination standard to be set should also be greater, thereby reducing the number of repeated monitoring processes to the greatest extent, so as to ensure that a monitoring process initiates a request for acquiring initial snapshot information to shorten the data transmission time. The preset snapshot information total number T0 can be divided into a first preset snapshot information total number T1 and a second preset snapshot information total number T2, T1 is set to 900, and T2 is set to 1200. The comparison process based on T, T1 and T2 is specifically as follows:

[0074] When T is less than or equal to T1, it is determined that the merging standard is set to 50. It can be understood that the minimum value of T at this time should meet the determination of enabling the merging standard. When T is greater than T1 and less than or equal to T2, it is determined that the merging standard is set to 30. When T is greater than T2, it is determined that the merging standard is set to 20, that is, when the number of repeated monitoring processes exceeds 20, the monitoring processes are merged. At this time, T is not infinite. It can be understood that T1 and T2 are not specifically limited in the embodiment of the application. In an optional embodiment, a new T1 = 950 and T2 = 1250 can also be set. As long as the merging standard is determined by comparing T with T1 and T2, the specific comparison process is referred to the above content. Meanwhile, the merging standard can also be set to other values meeting the requirements, for example, in the case of T less than or equal to T1, it can also be set to 45.

[0075] Further, the process of determining the optimization of the transmission environment comprises: counting the number of the initial snapshot information sent by the monitoring manager at the same time within a unit time and recording it as a snapshot concurrency; determining to increase the compression rate of the initial snapshot information based on the comparison result of the snapshot concurrency and a preset snapshot concurrency, and the increase range of the compression rate is in a positive correlation with the snapshot concurrency.

[0076] Specifically, in the embodiment, the snapshot concurrency D is used to determine the compression rate of the initial snapshot information, so as to reduce the data volume, reduce the bandwidth occupation, and thus optimize the transmission environment to reduce the data transmission time. The preset snapshot concurrency D0 can be divided into a first preset snapshot concurrency D1 and a second preset snapshot concurrency D2. D1 = 40 and D2 = 55 are set. The comparison process based on D, D1 and D2 is as follows:

[0077] If D is less than or equal to D1 and greater than 20, it is determined to increase by 10% on the basis of the original compression rate. If D is greater than D1 and less than or equal to D2, it is determined to increase by 15% on the basis of the original compression rate. If D is greater than D2, it is determined to increase by 20% on the basis of the original compression rate. It can be understood that the increase range of the compression rate is not infinite, and needs to meet the requirement of ensuring the integrity of the initial snapshot information. The increase range can also be set to other values meeting the requirements, for example, in the case of D greater than D2, the increase range is 18%. It can be understood that D1 and D2 are not specifically limited in the embodiment of the application. In an optional embodiment, a new D1 = 35 and D2 = 50 can also be set. As long as the increase range of the compression rate is determined by comparing D with D1 and D2, the specific comparison process is referred to the above content.

[0078] Further, after the comparison of the initial snapshot information and the change snapshot information is completed, the monitoring process stores the change snapshot information into the monitoring manager and replaces it with new initial snapshot information as the object of the next comparison. In this way, the snapshot information in the monitoring manager can be continuously updated to ensure that the latest snapshot information is used as the comparison object in each comparison process.

[0079] Further, when the log system captures multiple file operation commands operating on each target folder, multiple monitoring processes are correspondingly generated, each monitoring process being connected with the monitoring manager; the paths where each target folder is located are obtained, and each path is sent to the monitoring manager, and the monitoring manager sends the initial snapshot information corresponding to each path to the corresponding monitoring process. Through a centralized monitoring manager, unified management and scheduling of monitoring tasks are realized, and the flexibility and scalability of the system are improved.

[0080] Further, the process of importing the change file information into the data warehouse includes: the monitoring manager triggers an asynchronous notification mechanism and notifies the monitoring process, and the monitoring process imports the change file information into the data warehouse; wherein the asynchronous notification mechanism is a pre-set notification rule, including a message queue or a network hook. When a file change is detected, the asynchronous notification mechanism is triggered, and the change file information is sent to the running monitoring process according to the pre-set notification rule, and then the change file information is imported into the data warehouse or the data lake through the monitoring process.

[0081] It can be understood that in the embodiments of the present application, any one of the preset parameters or critical parameters is not specifically limited, and the above values are not limited thereto. A person skilled in the art can adjust the preset parameters or critical parameters according to actual needs or analysis of historical data or device usage.

[0082] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but a person skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. A person skilled in the art can make equivalent changes or replacements to related technical features without departing from the principles of the present application, and the technical solutions after the changes or replacements will fall within the protection scope of the present application.

[0083] The above description is only the preferred embodiments of the present application and is not intended to limit the present application; for those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for intelligent server listening for FTP files, characterized in that, The method comprises the following steps: initial snapshot processing is performed on each target folder in the FTP file server to obtain corresponding initial snapshot information and store the initial snapshot information in a monitoring manager; the FTP file server is connected with a user terminal, a login event of the user is monitored in real time through a log system in the FTP file server, and after the login event of the user is monitored, a file operation command executed by the user is obtained through the log system; a monitoring process is generated based on the file operation command executed in the target folder, and change snapshot processing is performed on the target folder to obtain corresponding change snapshot information; the monitoring process obtains the corresponding initial snapshot information from the monitoring manager and compares the initial snapshot information with the change snapshot information to determine whether a file change occurs in the corresponding target folder, and after it is determined that a file change occurs, change file information is imported into a data warehouse; a time length from when a request for obtaining the corresponding initial snapshot information is sent to when the initial snapshot information sent by the monitoring manager is received is calculated as a data transmission time length for each monitoring process, and whether the running process of the monitoring manager is qualified is determined based on each data transmission time length obtained in a preset running period; when it is determined that the running process is not qualified, corresponding instructions are determined based on each data transmission time length, the running memory of the monitoring manager is increased based on the instructions, repeated monitoring processes are combined, the initial snapshot information in the monitoring manager is updated, or the transmission environment is optimized; the process of optimizing the transmission environment comprises the following steps: the number of initial snapshot information sent by the monitoring manager at the same time in a unit time is counted as a snapshot concurrency number; based on the comparison result of the snapshot concurrency number and a preset snapshot concurrency number, the compression rate of the initial snapshot information is increased, and the increase amplitude of the compression rate is in a positive correlation with the snapshot concurrency number.

2. The method for intelligent monitoring of a server for FTP files according to claim 1, wherein, The process of determining whether the running process of the monitoring manager is qualified based on each data transmission time length corresponding to each monitoring process obtained in the preset running period comprises the following steps: each data transmission time length is counted and an average value is calculated to obtain a time length average value; whether the running process of the monitoring manager is qualified is determined based on the comparison result of the time length average value and a preset time length average value, or the total number of monitoring processes generated in the preset running period is used to re-determine whether the running process of the monitoring manager is qualified; when it is determined that the running process is not qualified, the reason for the unqualification is determined according to the difference between the time length average value and the preset time length average value.

3. The method for intelligent monitoring of a server for FTP files according to claim 2, wherein, The process of re-determining whether the running process of the monitoring manager is qualified based on the total number of monitoring processes generated in the preset running period comprises the following steps: the total number of monitoring processes is counted as a process total number; whether the running memory of the monitoring manager is increased is determined based on the comparison result of the process total number and a critical process total number.

4. The method for intelligent monitoring of a server for FTP files according to claim 3, wherein, The process of determining whether the running memory of the monitoring manager is increased based on the comparison result of the process total number and the critical process total number comprises the following steps: the process total number in each historical running period is counted, and a variance is calculated based on each process total number to obtain a process variance; The running memory is increased based on a comparison result of the process variance and a preset process variance, and the increase range of the running memory is negatively correlated with the process variance.

5. The method for intelligent monitoring of a server for FTP files according to claim 2, wherein, The process of determining the reason for disqualification based on the difference between the time length average and the preset time length average includes: calculating the difference between the time length average and the preset time length average and recording it as a time length average difference; determining the reason for disqualification based on a comparison result of the time length average difference and a preset time length average difference; generating a corresponding instruction based on the reason, determining to merge the repeated monitoring processes, determining to update the initial snapshot information in the monitoring manager, or determining to optimize the transmission environment based on the instruction.

6. The method for intelligent monitoring of a server for FTP files according to claim 5, wherein, The process of determining to merge the repeated monitoring processes includes: counting the total number of the initial snapshot information in the monitoring manager within the preset running period and recording it as a snapshot information total number; determining a merging standard based on a comparison result of the snapshot information total number and a preset snapshot information total number, and the merging standard is negatively correlated with the snapshot information total number; wherein, when the number of repeated monitoring processes is greater than or equal to the merging standard, the monitoring processes are merged.

7. The method for intelligent monitoring of a server for FTP files according to claim 1, wherein, After the comparison of the initial snapshot information and the changed snapshot information is completed, the monitoring process stores the changed snapshot information in the monitoring manager and replaces it with new initial snapshot information as the object for the next comparison.

8. The method for intelligent monitoring of a server for FTP files according to claim 1, wherein, When the log system captures multiple file operation commands operating on each target folder, multiple monitoring processes are generated correspondingly, and each monitoring process is connected with the monitoring manager. The paths of each target folder being operated are obtained, and each path is sent to the monitoring manager, and the monitoring manager sends the initial snapshot information corresponding to each path to the corresponding monitoring process.

9. The method for intelligent monitoring of a server for FTP files according to claim 1, wherein, The process of importing the changed file information into the data warehouse includes: The monitoring manager triggers an asynchronous notification mechanism and notifies the monitoring process, and the monitoring process imports the changed file information into the data warehouse. The asynchronous notification mechanism is a preset notification rule, including a message queue or a network hook.

Images