Cross-network log processing method and device, storage medium and program product

Abstract

Embodiments of the present disclosure disclose a cross-network log processing method and device, a storage medium and a program product. The method comprises: receiving logs from a first network in real time; storing the logs in a first log queue when the storage capacity of the first log queue meets a condition; creating a second log queue to store the logs when the storage capacity of the first log queue reaches a capacity threshold and there is no second log queue; writing the stored logs into a preset file through a first consumption thread; creating a second consumption thread to write the logs in the second log queue into the file when the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate shows a growth trend; and uploading the file to a second network after the file writing is completed. The method can make effective adjustment when the generation rate of logs fluctuates, and realize stable operation of log writing and uploading.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of network security, and particularly relates to a cross-network log processing method and device, a storage medium and a program product. BACKGROUND

[0002] In modern information systems, log recording is a key component of system operation and security audit. For some projects with strict security requirements, especially in scenarios requiring information isolation between intranet and extranet, data transmission is usually realized through a network gateway, that is, logs generated by one network are uploaded to another network through the network gateway. However, the generation rate of logs may fluctuate significantly, affecting transmission. SUMMARY

[0003] Therefore, the present disclosure provides a cross-network log processing method and device, a storage medium and a program product, which can make effective adjustments when the generation rate of logs fluctuates, and realize stable operation of log writing and uploading.

[0004] In a first aspect, the present disclosure provides a cross-network log processing method, which adopts the following technical scheme:

[0005] receiving logs from a first network in real time;

[0006] storing the logs in a first log queue when the storage capacity of the first log queue meets a condition;

[0007] creating a second log queue to store the logs when the storage capacity of the first log queue reaches a capacity threshold and there is no second log queue;

[0008] writing the stored logs into a preset file through a first consumption thread;

[0009] creating a second consumption thread to write logs in the second log queue into the file when the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate shows a growth trend;

[0010] uploading the file to a second network after the file writing is completed.

[0011] Optionally, when the storage capacities of the first log queue and the existing second log queue all reach the capacity threshold, a new second log queue is created to store logs.

[0012] Optionally, when the second log queue has not stored new logs within a preset expiration time, the second log queue is destroyed with delay.

[0013] Optionally, the writing of the stored logs into the preset file through the first consumption thread comprises:

[0014] The first consumer thread preferentially writes the logs in the first log queue into the file;

[0015] If the first log queue is empty, the second log queue is accessed in the order of creation. If the accessed second log queue contains logs but no corresponding second consumer thread, the first log queue writes the logs in the accessed second log queue into a file.

[0016] Optionally, each second log queue has a rate threshold;

[0017] When there are no logs in the second log queue having the second consumer thread and the current log receiving rate is less than the corresponding rate threshold, the second consumer thread is closed.

[0018] Optionally, when the number of logs written into the file reaches a preset number threshold, or the writing time of the file reaches a preset time threshold, it is determined that the writing of the file is completed.

[0019] Optionally, when the file writing is completed and the file queue is not empty, a new file is created for writing the log.

[0020] Optionally, after the file is written, uploading the file to the second network includes:

[0021] Uploading the file to the second network through a preset file queue and upload thread group;

[0022] The first log queue is a permanent log queue, and the second log queue is a temporary log queue; the first consumer thread is a permanent consumer thread, and the second consumer thread is a temporary consumer thread.

[0023] Optionally, uploading the file to the second network through a preset file queue and upload thread group includes:

[0024] Within a preset time interval, obtaining the production rate and consumption rate of the file queue at each time step;

[0025] Based on the production rate and the consumption rate, obtain the ideal number of threads required for each time step;

[0026] When the time interval is reached, adjusting the current number of threads of the upload thread group based on the ideal number of threads required for each time step;

[0027] Adding the path of the written file to the file queue, and the upload thread group uploading the corresponding file to the second network based on the path;

[0028] The calculation formula of the ideal thread number is:

[0029] L t =S t (X t / l t );

[0030] Wherein, t is the time step; L t is the ideal thread number at the time step; S t is the production rate of the file queue at the time step; X t is the consumption rate of the file queue at the time step; l t is the actual thread number of the upload thread group at the time step.

[0031] In a second aspect, the embodiments of the present disclosure further provide a cross-network log processing system, which adopts the following technical scheme:

[0032] The log real-time receiving module is configured to receive logs from the first network in real time;

[0033] The log storage queue module is configured to store the logs in the first log queue when the storage capacity of the first log queue meets the condition;

[0034] The log queue creation module is configured to create a second log queue to store the logs when the storage capacity of the first log queue reaches the capacity threshold and there is no second log queue;

[0035] The first log writing module is configured to write the stored logs into a preset file through a first consumption thread;

[0036] The second log writing module is configured to create a second consumption thread to write the logs in the second log queue into the file when the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate shows a growth trend;

[0037] The file upload network module is configured to upload the file to the second network after the file writing is completed.

[0038] In a third aspect, the embodiments of the present disclosure further provide a computer device, which adopts the following technical scheme:

[0039] The computer device comprises:

[0040] at least one processor; and

[0041] a memory connected in communication with the at least one processor; wherein

[0042] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the cross-network log processing method described in any one of the preceding embodiments.

[0043] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium storing computer instructions for causing a computer to perform the cross-network log processing method described in any one of the preceding embodiments.

[0044] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product comprising computer programs / instructions for implementing the steps of the method described in any one of the preceding embodiments when executed by a processor.

[0045] The cross-network log processing method provided by the embodiments of the present disclosure can timely complete the storage of logs through the dynamic management of the first log queue and the second log queue, effectively avoid the loss of log data, and ensure the integrity and continuity of logs. The storage strategy is flexibly adjusted according to the change of the amount of logs, and a new second log queue is created to adapt to different log receiving rates. The capacity monitoring and allocation of the second log queue enable the storage resources to be reasonably utilized and reduce unnecessary waste of resources. This real-time receiving and processing of logs and judgment according to the capacity threshold can quickly respond to the change of log flow and improve the processing efficiency of the system. Further, through the division of the first consumption thread and the second consumption thread, the pressure of the system under high load can be reduced, and the stability of log writing is ensured. The uploading thread group uploads the log file to the second network, enhancing the security and backup capability of log data and preventing data loss. It can be seen that this method can effectively adapt to the fluctuation of the generation rate of logs and make effective adjustments when the generation rate fluctuates. Moreover, the use of multiple queues to decouple the writing file and the uploading file improves the concurrent processing capability, realizes higher system flexibility and efficiency, and facilitates monitoring and management.

[0046] The above description is only a summary of the technical solutions of the present disclosure. In order to more clearly understand the technical means of the present disclosure, the contents of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0048] Figure 1 A flowchart of a cross-network log processing method provided by an embodiment of the present disclosure is shown in FIG. 1.

[0049] Figure 2 An architecture diagram of a cross-network log processing system provided by an embodiment of the present disclosure is shown in FIG. 2.

[0050] Figure 3 A flowchart of a delay destruction method provided by an embodiment of the present disclosure is shown in FIG. 3.

[0051] Figure 4 A flowchart of a preset policy provided by an embodiment of the present disclosure is shown in FIG. 4.

[0052] Figure 5 A flowchart of an upload thread group dynamic adjustment provided by an embodiment of the present disclosure is shown in FIG. 5.

[0053] Figure 6 A principle block diagram of a cross-network log processing system provided by an embodiment of the present disclosure is shown in FIG. 6.

[0054] Figure 7 A structure diagram of a computer device provided by an embodiment of the present disclosure is shown in FIG. 7. DETAILED DESCRIPTION

[0055] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0056] It should be apparent that the following describes only some embodiments of the present disclosure and not all embodiments of the present disclosure. The present disclosure can be implemented or applied in other different specific embodiments, and each detail described in the specification can be modified or changed based on different views and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present disclosure.

[0057] It is important to note that the various aspects described herein can be implemented in a wide variety of forms, and that any particular structure and / or function described herein is merely illustrative. Based on the teachings herein one skilled in the art will appreciate that an aspect described herein can be implemented independently of any other aspects and that an aspect described herein can be implemented both as any number of software and / or hardware structures and as any number of processes and / or operations. An aspect can be implemented as a system, method, apparatus, program, software, and / or any combination of the foregoing. One skilled in the art will appreciate that an aspect can be implemented by any number of hardware and / or software components and that the aspects described herein need not include all of the components described.

[0058] It is also important to note that the present disclosure can be carried out by more than one process. Further, it is important to note that some of the procedures can be carried out in an order other than the described, which can depend on the implementation. Various modifications and changes can be made as would be obvious to a person skilled in the art having the benefit of this disclosure, and it is intended to embrace all such modifications and changes and, accordingly, the above description to be construed in an illustrative, and not a restrictive, sense.

[0059] In addition, in the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, etc., to provide a thorough understanding of examples. One skilled in the relevant art will recognize, however, that the described aspects can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth.

[0060] Reference Figure 1 The present disclosure provides a cross-network log processing method, applied to a network gateway, comprising the following steps:

[0061] S1: receiving logs from a first network in real time;

[0062] S2: determining whether the storage capacity of the first log queue reaches a capacity threshold; if not, performing S3; if yes, performing S4;

[0063] S3: storing logs in the first log queue;

[0064] S4: determining whether there is a second log queue; if not, performing S5; if yes, performing S6;

[0065] S5: creating a second log queue to store logs;

[0066] S6: determining whether the storage capacity of the existing second log queue reaches a capacity threshold according to the creation order; if there is a second log queue whose storage capacity does not reach the capacity threshold, performing S7; if the storage capacity of all second log queues reaches the capacity threshold, performing S8;

[0067] S7: storing logs in the second log queue whose storage capacity does not reach the capacity threshold;

[0068] S8: create a new second log queue to store logs;

[0069] S9: write the stored logs into a preset file according to a preset strategy by a first consumption thread;

[0070] S10: when the storage capacity of the second log queue reaches a capacity threshold and the current log receiving rate shows a growing trend, create a second consumption thread to write the logs in the second log queue into a file;

[0071] S11: after the file writing is completed, upload the file to a second network through a preset file queue and a group of upload threads.

[0072] Among them, the first log queue is a resident log queue, and the second log queue is a temporary log queue; the first consumption thread is a resident consumption thread, and the second consumption thread is a temporary consumption thread.

[0073] Referring to Figure 2 The architecture diagram of the cross-network log processing system is shown, which is for a first network and a second network, either of which is an external network and the other is an internal network. The gateway receives logs from the first network in real time, processes the logs, and uploads the logs to the second network in the form of a file.

[0074] The cross-network log processing method of the present disclosure can timely complete the storage of logs through the dynamic management of the first log queue and the second log queue, effectively avoid the loss of log data, and ensure the integrity and continuity of the logs. According to the change of the log amount, the storage strategy is flexibly adjusted, and a new second log queue is created to adapt to different log receiving rates. The capacity monitoring and allocation of the second log queue make the storage resources be reasonably utilized, reducing unnecessary resource waste. This real-time receiving and processing of logs and judgment according to the capacity threshold can quickly respond to the change of log flow and improve the processing efficiency of the system. Further, through the division of labor of the first consumption thread and the second consumption thread, the pressure of the system under high load can be reduced, and the stability of log writing can be ensured. Through the group of upload threads, the log file is uploaded to the second network, enhancing the security and backup ability of the log data and preventing data loss. It can be seen that this method can effectively adapt to the fluctuation of the generation rate of logs and make effective adjustments when the generation rate fluctuates. Moreover, the writing of files and the uploading of files are decoupled by using multiple queues, improving the concurrent processing capability, realizing higher system flexibility and efficiency, and facilitating monitoring and management.

[0075] In summary, through the efficient log management and processing mechanism, the reliability, flexibility and security of the system are improved.

[0076] In S1-S8, the gateway is configured with a first log queue and preferentially attempts to store received logs in the first log queue. If the storage capacity of the first log queue does not reach the capacity threshold, the storage operation succeeds. If the storage capacity of the first log queue reaches the capacity threshold, the storage operation fails and a check is performed to determine whether a second log queue already exists. If not, a second log queue is created to store the logs. If a second log queue exists, existing second log queues are checked in the order in which they were created (referred to as the creation order). The check is complete when the storage capacity of the second log queue being checked does not reach the capacity threshold or when all existing second log queues have been checked. If any second log queue is found to have a storage capacity that does not reach the capacity threshold, the log is stored in that second log queue. If the storage capacity of all second log queues reaches the capacity threshold, a new second log queue is created to store the logs.

[0077] The above method of sequentially trying the first and second log queues is equivalent to setting a priority for each log queue. The first log queue has the highest priority and requires logs to be written first. The second log queue has a lower priority and decreases in order of creation. The logs can only be stored in the later created log queue after the first created log queue meets the conditions. This method can reduce CPU switching between consumer threads and minimize the creation of multiple log queues, thereby reducing the number of consumer threads used, using the least number of consumer threads to process logs, and reducing the possibility of CPU switching between consumer threads.

[0078] The capacity threshold is 50% of the maximum log write volume per second in single-threaded mode. By setting the capacity threshold, 0.5 seconds can be reserved to cope with consumption rate fluctuations when switching between consumer threads and writing files. The maximum log write volume per second in single-threaded mode is calculated through simulated writes after the service is started.

[0079] Furthermore, an expiration time is set. When the second log queue does not store new logs within the expiration time, the second log queue is delayed and destroyed. Figure 3 The flowchart of the delayed destruction method shown in the figure includes the following steps:

[0080] S12: Determine in real time whether each second log queue stores a new log within the expiration time; if so, execute S12; if not, execute S13;

[0081] S12: Determine that the second log queue is a non-expired queue;

[0082] S13: Determine that the second log queue is an expired queue;

[0083] S14: Query the unexpired queue closest to the expired queue creation time, determine the second log queue quantity between the two, recorded as the intermediate queue number; wherein the first log queue is the unexpired queue;

[0084] S15: The greater the second log queue quantity, the longer the delay destruction time set for the expired queue;

[0085] S16: Determine whether the expired queue stores new logs within the delay destruction time; if yes, execute S17; if no, execute S18;

[0086] S17: Modify the expired queue to the unexpired queue;

[0087] S18: Destroy the expired queue.

[0088] Wherein, the expiration time and the delay destruction time are both time ranges, wherein the calculation formula of the delay destruction time is as follows:

[0089] T i = T A + T B *C;

[0090] In the formula, i is the number of the second log queue, the number is set according to the creation order; T i is the delay destruction time of the expired queue numbered i; T A is a preset base time, which is a fixed value; T B is a preset increase value, which is also a fixed value; C is the intermediate queue number corresponding to the expired queue numbered i, specifically referring to the number of the second log queue created between the expired queue numbered i and the unexpired queue closest to the creation time of the expired queue.

[0091] In S9, the gateway is provided with a first consumption thread, which writes the stored logs into a preset file according to a preset strategy, referring to the flowchart of the preset strategy shown in Figure 4 The method for the first consumption thread to process logs includes the following steps:

[0092] S91: Determine whether the first log queue is empty; if no, execute S92; if yes, execute S93;

[0093] S92: The first consumption thread writes the logs in the first log queue into a preset file;

[0094] S93: Access the second log queue according to the creation order, determine whether there are logs in the second log queue and no corresponding second consumption thread; if yes, execute S94; if no, execute S95;

[0095] S94: The first log queue writes the logs in the second log queue to be accessed into a file;

[0096] S95: The count value is added by 1, and the initial value of the count value is 0;

[0097] S96: It is determined whether the count value reaches a preset value; if yes, S97 is executed; if no, S91 is returned;

[0098] S97: The first log queue enters sleep, and the sleep time is recorded;

[0099] S98: It is determined whether the sleep time reaches a preset sleep threshold; if yes, S97 is returned to continue sleep; if no, S99 is executed;

[0100] S99: The first log queue is woken up, and the sleep time and the count value are cleared, and S91 is returned.

[0101] Through the above preset strategy, the first log queue obtains the maximum use value, thereby reducing the number of second consumer threads to be created, reducing system consumption, improving the efficiency of thread management and scheduling, and at the same time, when there is no new log for a long time, the first log queue enters sleep to save CPU resources, further reducing system consumption.

[0102] In S10, when the storage capacity of the second log queue reaches the capacity threshold, the current log receiving rate and the previous log receiving rate are obtained, wherein the current log receiving rate is equal to the number of logs received by the gateway in the current time step, and the previous log receiving rate is equal to the number of logs received by the gateway in the previous time step. The present disclosure sets 1 time step as 1 second. If the difference between the current log receiving rate and the previous log receiving rate is positive, it indicates that the current log receiving rate is in a growth trend; if the difference is negative, it indicates that the current log receiving rate is in a downward trend; if the difference is 0, it indicates that the current log receiving rate is in a stable trend. If it is in a growth trend, a second consumer thread is created to write the logs in the second log queue into a file

[0103] When the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate is in a growth trend, a second consumer thread dedicated to the second log queue is created, and the second consumer thread is responsible for writing the logs in the second log queue into a file. If the storage capacity of the second log queue does not reach the capacity threshold, or the current log receiving rate is not in a growth trend, a second consumer thread dedicated to the second log queue does not need to be created.

[0104] Further, each second log queue has an independent rate threshold, which refers to the current log receiving rate saved when the second log queue is created. When there is no log in the second log queue with the second consumption thread and the current log receiving rate is less than the corresponding rate threshold, the second consumption thread is closed.

[0105] The single-thread writing log is low in efficiency and may cause blocking, and the way of using multi-thread to write log into the same file can improve the writing speed, but also brings the problem of lock competition and frequent resource creation, which may cause the system performance to decline. The method of multi-thread writing log into multiple files will produce too many files when the log receiving rate is low, resulting in waste of server resources. The cross-network log processing method of the present disclosure can reduce the number of threads created when the log amount is small, save system resources as much as possible while not affecting log processing, and increase the number of second log queues and threads created when the log amount is large, so as to avoid waste of system resources as much as possible while ensuring efficient log processing.

[0106] In S11, in one specific embodiment, the method for determining whether the file writing is completed, comprising: when the number of logs written into the file reaches a preset number threshold, or the writing time of the file reaches a preset time threshold, determining that the file writing is completed; when the number of logs written into the file does not reach the preset number threshold, and the writing time of the file does not reach the preset time threshold, determining that the file writing is not completed. The writing time of the file refers to the time when the file is written by the log, and can also be understood as the time when the file receives the log. The number threshold can be set to 1000.

[0107] In another specific embodiment, the method for determining whether the file writing is completed, comprising: first determining whether the number of logs written into the file reaches a preset number threshold; if yes, determining that the file writing is completed; if no, determining whether the writing time of the file reaches a preset time threshold; if yes, determining that the file writing is completed; if no, the file writing is not completed.

[0108] Further, the gateway is provided with a file queue and an upload thread group, the upload thread group includes a plurality of upload threads, when the file writing is completed, the path of the file is added to the file queue, the upload threads in the upload thread group constantly obtain the path from the file queue, find the corresponding file according to the path and upload the file to the second network, after the uploading is completed, it is determined whether the file queue is empty; if yes, the upload threads in the upload thread group wait; if no, the uploading operation is continued.

[0109] It is noted that the cross-network log processing method described above can write logs into files in the order of receiving by the gateway as much as possible, so as to ensure that the logs can be arranged in time sequence and facilitate management. When the file writing is completed and the file queue is not empty, a new file is created and the newly received logs are written into the new file. Compared with the number of received logs, the number of generated files is much smaller, and usually only one file queue can be used to realize the operation of the system.

[0110] However, the number of upload threads in the upload thread group can be dynamically adjusted according to the number of files, so as to save system resources while meeting transmission efficiency. In an embodiment, every interval of a preset time, the production rate and the consumption rate of the file queue are obtained. If the production rate is greater than the consumption rate, it indicates that the upload rate is small, and the current number of threads of the upload thread group is increased. If the production rate is less than the consumption rate, it indicates that the file consumption efficiency is relatively high, and the current number of threads of the upload thread group is reduced to avoid resource waste. If the production rate is equal to the consumption rate, it indicates that the file generation and consumption rates are relatively balanced, and the current number of threads of the upload thread group does not need to be adjusted.

[0111] In the embodiment, the production rate is equal to the number of successfully added paths of the file writing completion to the file queue in the current time step, and the consumption rate is equal to the number of successfully uploaded files by the upload thread group to the second network through the file queue in the current time step.

[0112] In the embodiment, the current production rate and the consumption rate can be used to judge the processing state of the file and make timely adjustment to the current number of threads of the upload thread group.

[0113] In another embodiment, the specific method for processing the upload thread group includes: obtaining the production rate and the consumption rate of the file queue in each time step within a preset time interval. The difference between the production rate and the consumption rate of each time step is obtained, the number of positive differences and the number of negative differences are counted, and if the number of positive differences is greater than the number of negative differences, it indicates that the upload rate is small, and the current number of threads of the upload thread group is increased. If the number of negative differences is greater than the number of positive differences, it indicates that the file consumption efficiency is relatively high, and the current number of threads of the upload thread group is reduced to avoid resource waste. If the number of positive differences is equal to the number of negative differences, it indicates that the file generation and consumption rates are relatively balanced, and the current number of threads of the upload thread group does not need to be adjusted.

[0114] In the embodiment, the production rate and the consumption rate in a period of time can be concentrated, and the file processing state in the period of time can be evaluated by the difference, and then more accurate adjustment of the current number of threads can be made.

[0115] In yet another embodiment, with reference to Figure 5The flowchart of the dynamic adjustment of the uploaded thread group is shown in the figure, and the specific method for processing the uploaded thread group comprises the following steps:

[0116] S111: In a preset time interval, the production rate and the consumption rate of the file queue at each time step are obtained;

[0117] S112: Based on the production rate and the consumption rate, the ideal thread number required at each time step is obtained;

[0118] S113: When the time interval is reached, the current thread number of the uploaded thread group is adjusted based on the ideal thread number required at each time step.

[0119] For example, the time interval is set to 2 minutes, and the time step is 1 second, then in each 2 minutes, the production rate and the consumption rate are obtained 120 times, and the ideal thread number is calculated 120 times. The current thread number of the uploaded thread group is equal to the average value of the ideal thread number required at each time step, therefore, the average value of the ideal thread number calculated 120 times is taken, and the current thread number of the uploaded thread group is adjusted to equal to the average value.

[0120] The calculation formula of the ideal thread number is:

[0121] L t =S t (X t / l t );

[0122] In the formula, t is the time step at that time, L t is the ideal thread number at the time step, S t is the production rate of the file queue at the time step, X t is the consumption rate of the file queue at the time step, and l t is the actual thread number of the uploaded thread group at the time step.

[0123] The embodiment can adapt to the current file processing scene as much as possible by calculating the ideal thread number in a period of time and then calculating the average value to find the uploaded thread number in the current most ideal state.

[0124] The above contents all adopt the method of delayed adjustment, that is, non-real-time adjustment, to avoid the instability of the fluctuation of the uploaded thread number, which causes large system burden. Through the method of adjusting the uploaded thread number at intervals, the large fluctuation of the uploaded thread number is avoided, and the timeliness of the adjustment is guaranteed as much as possible, so as to adjust the processing efficiency of the file to the best as soon as possible.

[0125] In summary, the cross-network log processing method of the present application overcomes the problems of the log receiving rate being large or small, the low efficiency of writing files, the waste of server resources, and the decoupling of writing and uploading. The method dynamically adjusts the use of server resources by monitoring the log receiving rate, reuses existing server resources to avoid resource waste, uses a queue to reduce the coupling between writing files and uploading files, greatly improves the efficiency of writing files and uploading files, and reasonably uses server resources.

[0126] With reference to Figure 6 The present disclosure provides a cross-network log processing system, comprising:

[0127] A log real-time receiving module 101 is configured to receive logs from a first network in real time.

[0128] A log storage queue module 102 is configured to store the logs in a resident log queue first log queue when the storage capacity of the resident log queue first log queue meets a condition.

[0129] A log queue creation module 103 is configured to create a temporary log queue second log queue to store the logs when the storage capacity of the resident log queue first log queue reaches a capacity threshold and there is no second log queue.

[0130] A first log writing module 104 is configured to write the stored logs into a preset file through a resident consumption thread first consumption thread.

[0131] A second log writing module 105 is configured to create a temporary consumption thread second consumption thread to write the logs in the temporary log queue second log queue into the file when the storage capacity of the temporary log queue second log queue reaches the capacity threshold and the current log receiving rate shows a growth trend.

[0132] A file uploading network module 106 is configured to upload the file to a second network after the file writing is completed.

[0133] The various variations and specific examples of the cross-network log processing method provided above are also applicable to the cross-network log processing system provided by the present disclosure. Through the foregoing detailed description of the cross-network log processing method, those skilled in the art can clearly understand the implementation method of the cross-network log processing system. For the sake of brevity of the specification, the implementation method of the cross-network log processing system will not be described in detail here.

[0134] A computer device according to an embodiment of the present disclosure includes a memory and a processor. The memory is configured to store non-transitory computer readable instructions. Specifically, the memory can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory, among others. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, among others.

[0135] The processor can be a central processing unit (CPU) or other form of processing unit that has data processing and / or instruction execution capabilities, and can control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is configured to execute the computer readable instructions stored in the memory, so that the computer device performs all or part of the steps of the cross-network log processing method according to the embodiments of the present disclosure.

[0136] Those skilled in the art will understand that, in order to solve the technical problem of how to obtain a good user experience effect, the present embodiment can also include well-known structures such as a communication bus, an interface, etc., which should also be included in the protection scope of the present disclosure.

[0137] As Figure 7 A structural schematic diagram of a computer device according to an embodiment of the present disclosure is shown. It shows a structural schematic diagram of a computer device suitable for use in implementing the computer device according to an embodiment of the present disclosure. Figure 7 The computer device shown is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0138] As Figure 7 As shown, the computer device can include a processor (such as a central processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) or loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0139] Generally, the following devices can be connected to the I / O interface: input devices including, for example, sensors or visual information acquisition devices, etc.; output devices including, for example, display screens, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. The communication devices can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Although Figure 7The computer device is shown with various devices, but it should be understood that not all of the shown devices are required to implement or be present. More or fewer devices can alternatively be implemented or present.

[0140] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the cross-network log processing method of embodiments of the present disclosure are performed.

[0141] Detailed descriptions of the present embodiments can refer to the corresponding descriptions of the previous embodiments, which will not be repeated here.

[0142] The computer-readable storage medium according to the embodiments of the present disclosure has non-transitory computer-readable instructions stored thereon. When the non-transitory computer-readable instructions are run by a processor, all or part of the steps of the cross-network log processing method of the embodiments of the present disclosure described above are performed.

[0143] The computer-readable storage medium described above includes, but is not limited to, optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or a removable hard disk), media with built-in rewritable non-volatile memory (e.g., a memory card), and media with built-in ROM (e.g., a ROM cartridge).

[0144] Detailed descriptions of the present embodiments can refer to the corresponding descriptions of the previous embodiments, which will not be repeated here.

[0145] The basic principles of the present disclosure are described above in conjunction with specific embodiments, but it should be noted that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples and are not limiting, and these advantages, advantages, effects, etc. cannot be considered as the must-have of each embodiment of the present disclosure. In addition, the above-mentioned specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above-mentioned details do not limit the present disclosure to the must-use of the above-mentioned specific details.

[0146] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0147] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0148] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0149] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0150] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0151] The foregoing description has been presented for the purposes of illustration and description. Furthermore, the description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although the various example aspects and embodiments have been described herein with regard to particular aspects and embodiments, those skilled in the art will recognize that certain modifications, changes, substitutions, additions and sub-combinations can be made without departing from the spirit of the disclosure.

Claims

1. A cross-network log processing method, characterized by, The method comprises: receiving logs from a first network in real time; storing the logs into a first log queue when the storage capacity of the first log queue meets a condition; creating a second log queue to store the logs when the storage capacity of the first log queue reaches a capacity threshold and there is no second log queue; writing the stored logs into a preset file by a first consumption thread; creating a second consumption thread to write the logs in the second log queue into the file when the storage capacity of the second log queue reaches the capacity threshold and a current log receiving rate is in a growth trend; uploading the file to a second network after the file writing is completed; delaying the destruction of the second log queue when the second log queue has not stored new logs within a preset expiration time; each second log queue has a rate threshold; closing the second consumption thread when there is no log in the second log queue with the second consumption thread and the current log receiving rate is less than the corresponding rate threshold.

2. The cross-network log processing method of claim 1, wherein, creating a new second log queue to store logs when the storage capacities of the first log queue and the existing second log queue all reach the capacity threshold.

3. The cross-network log processing method of claim 1, wherein, The writing of the stored logs into the preset file by the first consumption thread comprises: the first consumption thread preferentially writes the logs in the first log queue into the file; if the first log queue is empty, the second log queues are accessed in the order of creation, and if there are logs in the accessed second log queue and there is no corresponding second consumption thread, the first log queue writes the logs in the accessed second log queue into the file.

4. The cross-network log processing method of claim 1, wherein, The file writing is determined to be completed when the number of logs written into the file reaches a preset number threshold or the writing time of the file reaches a preset time threshold.

5. The cross-network log processing method of claim 1, wherein, When the file writing is completed and the file queue is not empty, a new file is created for writing logs.

6. The cross-network log processing method of claim 1, wherein, The uploading of the file to the second network after the file writing is completed comprises: uploading the file to the second network by a preset file queue and an uploading thread group. The first log queue is a permanent log queue, the second log queue is a temporary log queue, the first consumption thread is a permanent consumption thread, and the second consumption thread is a temporary consumption thread.

7. The cross-network log processing method of claim 6, wherein, The uploading of the file to the second network by the preset file queue and the uploading thread group comprises: obtaining the production rate and the consumption rate of the file queue at each time step within a preset time interval; obtaining the ideal number of threads required at each time step based on the production rate and the consumption rate; adjusting the current number of threads of the uploading thread group based on the ideal number of threads required at each time step when the time interval is reached; adding the path of the file written into the file queue, and the uploading thread group uploads the corresponding file to the second network based on the path; wherein the calculation formula of the ideal number of threads is: L t =S t (X t / l t ); where t is the current time step; L t is the ideal number of threads at the current time step; S t is the production rate of the file queue at the current time step; X t is the consumption rate of the file queue at the current time step; l t is the actual number of threads of the upload thread group at the current time step.

8. A computer apparatus, comprising: The computer device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the cross-network log processing method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the cross-network log processing method of any one of claims 1-7.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions, when executed by a processor, implement the steps of the method of any one of claims 1-7.

Images