Cross-network log processing method and device, storage medium and program product

Abstract

The embodiment of the invention discloses a cross-network log processing method and device, a storage medium and a program product. The method comprises the following steps: receiving a log from a first network in real time; when the storage capacity of the first log queue meets the condition, storing the log into the first log queue; when the storage capacity of the first log queue reaches a capacity threshold value and no second log queue exists, creating the second log queue to store the log; writing the stored log into a preset file through the first consumption thread; when the storage capacity of the second log queue reaches the capacity threshold value and the current log receiving rate is in an increasing trend, creating a second consumption thread to write logs in the second log queue into the file; and after the file is written, uploading the file to a second network. According to the method, effective adjustment can be made when the generation rate of the log fluctuates, and stable operation of log writing and uploading is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of network security technology, and in particular to a cross-network log processing method, device, storage medium, and program product. Background Art

[0002] In modern information systems, logging is a key component of system operation and security auditing. For some projects with strict security requirements, especially those that require information isolation between the intranet and the extranet, data transmission is usually achieved through a gateway, that is, the logs generated by one network are uploaded to another network through the gateway. However, the log generation rate may fluctuate significantly, affecting transmission. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a cross-network log processing method, device, storage medium, and program product, which can make effective adjustments when the log generation rate fluctuates, thereby achieving stable operation of log writing and uploading.

[0004] In a first aspect, the embodiments of the present disclosure provide a cross-network log processing method, which adopts the following technical solutions:

[0005] Receive logs from the first network in real time;

[0006] When the storage capacity of the first log queue meets the condition, storing the log in the first log queue;

[0007] When the storage capacity of the first log queue reaches the capacity threshold and there is no second log queue YY+242252P

[0008] When the log is in the queue, a second log queue is created to store the log;

[0009] Write the stored logs to a preset file through the first consumer thread;

[0010] When the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate shows an increasing trend, a second consumer thread is created to write the logs in the second log queue into the file;

[0011] After the file is written, the file is uploaded to the second network.

[0012] Optionally, when the storage capacity of the first log queue and the existing second log file both reach a capacity threshold, a new second log queue is created to store logs.

[0013] Optionally, when the second log queue does not store any new log within a preset expiration time, the second log queue is destroyed with delay.

[0014] Optionally, writing the stored log into a preset file through the first consumer thread includes:

[0015] The first consumer thread preferentially writes the logs in the first log queue into the file;

[0016] If the first log queue is empty, the second log queue is accessed in the order of creation. If the accessed second log queue contains logs but has no corresponding second consumer thread, the first log queue writes the logs in the accessed second log queue into a file.

[0017] Optionally, each second log queue has a rate threshold;

[0018] When there is no log in the second log queue having the second consumer thread, and the current log receiving rate is less than the corresponding rate threshold, the second consumer thread is closed.

[0019] Optionally, when the number of logs written into the file reaches a preset number threshold, or the writing time of the file reaches a preset time threshold, it is determined that the writing of the file is completed.

[0020] Optionally, when the file writing is completed and the file queue is not empty, a new file is created for writing the log.

[0021] Optionally, after the writing of the file is completed, uploading the file to the second network includes:

[0022] Uploading the file to the second network through a preset file queue and upload thread group;

[0023] The first log queue is a permanent log queue, and the second log queue is a temporary log queue; the first consumer thread is a permanent consumer thread, and the second consumer thread is a temporary consumer thread.

[0024] Optionally, uploading the file to the second network through a preset file queue and an upload thread group includes:

[0025] Within a preset time interval, obtaining the production rate and consumption rate of the file queue at each time step;

[0026] Based on the production rate and the consumption rate, obtain the ideal number of threads required for each time step;

[0027] When the time interval is reached, adjusting the current number of threads of the upload thread group based on the ideal number of threads required for each time step;

[0028] Adding the path of the written file to the file queue, and the upload thread group uploading the corresponding file to the second network based on the path;

[0029] The calculation formula for the ideal number of threads is:

[0030] L t =S t (X t / l t );

[0031] Among them, t is the time step at that time; L t is the ideal number of threads at the time step; S t is the production rate of the file queue at the current time step; X t The consumption of the file queue at the current time step is YY+242252P

[0032] Rate; l t The actual number of threads in the upload thread group at the current time step.

[0033] In a second aspect, the embodiment of the present disclosure also provides a cross-network log processing system, which adopts the following technical solution:

[0034] A log real-time receiving module, used for receiving logs from the first network in real time;

[0035] A log storage queue module, used for storing the log in the first log queue when the storage capacity of the first log queue meets the condition;

[0036] A log queue creation module, used for creating a second log queue to store the log when the storage capacity of the first log queue reaches a capacity threshold and there is no second log queue;

[0037] A first log writing module, used for writing the stored logs into a preset file through a first consumer thread;

[0038] A second log writing module is used to create a second consumption thread to write the logs in the second log queue into the file when the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate shows an increasing trend;

[0039] The file upload network module is used to upload the file to the second network after the file is written.

[0040] In a third aspect, the embodiments of the present disclosure further provide a computer device, which adopts the following technical solution:

[0041] The computer device comprises:

[0042] at least one processor; and,

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the YY+242252P

[0045] Any of the above cross-network log processing methods.

[0046] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned cross-network log processing methods.

[0047] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0048] The cross-network log processing method provided by the embodiment of the present disclosure can complete the storage of logs in time through the dynamic management of the first log queue and the second log queue, effectively avoid the loss of log data, and ensure the integrity and continuity of the log. According to the change of the log volume, the storage strategy is flexibly adjusted to support the creation of a new second log queue to adapt to different log receiving rates. The capacity monitoring and allocation of the second log queue make the storage resources reasonably utilized and reduce unnecessary resource waste. This real-time reception and processing of logs, and according to the judgment of the capacity threshold, can quickly respond to the change of log flow and improve the processing efficiency of the system. Further, through the division of labor between the first consumer thread and the second consumer thread, the pressure of the system under high load can be reduced and the stability of log writing can be guaranteed. The log file is uploaded to the second network through the upload thread group, which enhances the security and backup capability of the log data and prevents data loss. It can be seen that the method can effectively adapt to the fluctuation of the log generation rate and make effective adjustments when the generation rate fluctuates. In addition, multiple queues are used to decouple the writing file and the uploading file, improve the concurrent processing capability, achieve higher system flexibility and efficiency, and facilitate monitoring and management.

[0049] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic diagram of a flow chart of a cross-network log processing method provided by an embodiment of the present disclosure;

[0052] Figure 2 An architecture diagram of a cross-network log processing system provided by an embodiment of the present disclosure;

[0053] Figure 3 A flowchart of a delayed destruction method provided by an embodiment of the present disclosure;

[0054] Figure 4 A schematic diagram of a flow chart of a preset strategy provided in an embodiment of the present disclosure;

[0055] Figure 5 A schematic diagram of a process for dynamically adjusting an upload thread group provided in an embodiment of the present disclosure;

[0056] Figure 6 A principle block diagram of a cross-network log processing system provided by an embodiment of the present disclosure;

[0057] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0059] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0060] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0061] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0062] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0063] Reference Figure 1 The present disclosure provides a cross-network log processing method, which is applied to a network gate, and includes the following steps:

[0064] S1: receiving logs from the first network in real time;

[0065] S2: Determine whether the storage capacity of the first log queue has reached the capacity threshold; if not, then YY+242252P

[0066] Execute S3; if yes, execute S4;

[0067] S3: Store the log in the first log queue;

[0068] S4: Determine whether the second log queue exists; if not, execute S5; if so, execute S6;

[0069] S5: Create a second log queue to store logs;

[0070] S6: Determine whether the storage capacity of the existing second log queues reaches the capacity threshold in the order of creation; if the storage capacity of any second log queue does not reach the capacity threshold, execute S7; if the storage capacity of all second log queues reaches the capacity threshold, execute S8;

[0071] S7: storing the log in a second log queue that has not reached the capacity threshold;

[0072] S8: Create a new second log queue to store logs;

[0073] S9: Writing the stored logs into a preset file according to a preset strategy through the first consumer thread;

[0074] S10: When the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate shows an increasing trend, a second consumer thread is created to write the logs in the second log queue into a file;

[0075] S11: After the file is written, the file is uploaded to the second network through a preset file queue and upload thread group.

[0076] Among them, the first log queue is a permanent log queue, and the second log queue is a temporary log queue; the first consumer thread is a permanent consumer thread, and the second consumer thread is a temporary consumer thread.

[0077] Reference Figure 2 The architecture diagram of the cross-network log processing system is shown. For the first network and the second network, either one is an external network and the other is an internal network. The network gate receives the logs from the first network in real time, processes the logs, and uploads the logs to the first network in the form of files.

[0078] Second network.

[0079] The cross-network log processing method disclosed in the present invention can complete the storage of logs in time through the dynamic management of the first log queue and the second log queue, effectively avoid the loss of log data, and ensure the integrity and continuity of the log. According to the change of the log volume, the storage strategy is flexibly adjusted to support the creation of a new second log queue to adapt to different log receiving rates. The capacity monitoring and allocation of the second log queue make the storage resources reasonably utilized and reduce unnecessary resource waste. This real-time reception and processing of logs, and according to the judgment of the capacity threshold, can quickly respond to the change of log flow and improve the processing efficiency of the system. Further, through the division of labor of the first consumer thread and the second consumer thread, the pressure of the system under high load can be reduced and the stability of log writing can be guaranteed. The log file is uploaded to the second network through the upload thread group, which enhances the security and backup capability of the log data and prevents data loss. It can be seen that the method can effectively adapt to the fluctuation of the generation rate of the log and make effective adjustments when the generation rate fluctuates. In addition, multiple queues are used to decouple the writing file and the uploading file, improve the concurrent processing capability, achieve higher system flexibility and efficiency, and facilitate monitoring and management.

[0080] In summary, this method improves the reliability, flexibility and security of the system through efficient log management and processing mechanism.

[0081] In S1-S8, the network gate is provided with a first log queue, and the received log is preferentially tried to be stored in the first log queue; if the storage capacity of the first log queue does not reach the capacity threshold, the storage operation is successful; if the storage capacity of the first log queue reaches the capacity threshold, the storage operation fails, and it is checked whether a second log queue already exists; if not, a second log queue is created for storing logs; if a second log queue exists, the existing second log queue is detected in the order of creation (referred to as the creation order), and the detection operation is completed when the storage capacity of the second log queue being detected does not reach the capacity threshold or all the existing second log queues are detected. If it is detected that the storage capacity of a second log queue does not reach the capacity threshold, the log is stored in the second log queue; if the storage capacity of all second log queues reaches the capacity threshold, a new second log queue is created for storing logs.

[0082] The above method of trying the first log queue and the second log queue in sequence is equivalent to setting a priority for each log queue, wherein the first log queue has the highest priority and needs to write logs first, the second log queue has a lower priority, and the priority decreases in the order of creation. The log queue created first needs to meet the conditions before the log can be stored in the log queue created later. This method can reduce the CPU switching between consumer threads, minimize the creation of multiple log queues, thereby reducing the number of consumer threads used, using the least consumer threads to process logs, and reducing the possibility of CPU switching between consumer threads.

[0083] The capacity threshold is 50% of the maximum log write volume per second in single-threaded mode. By setting the capacity threshold, 0.5 seconds can be reserved to cope with consumption rate fluctuations when switching between consumer threads and writing files. The maximum log write volume per second in single-threaded mode is calculated through simulated writing after the service is started.

[0084] Furthermore, an expiration time is set. When the second log queue does not store new logs within the expiration time, the second log queue is delayed and destroyed. Figure 3 The flowchart of the delayed destruction method shown in the figure determines whether to delay the destruction of the second log queue and the specific method of delayed destruction, including the following steps:

[0085] S12: Determine in real time whether each second log queue stores a new log within the expiration time; if so, execute S12; if not, execute S13;

[0086] YY+242252P

[0087] S12: Determine that the second log queue is a non-expired queue;

[0088] S13: Determine that the second log queue is an expired queue;

[0089] S14: querying the unexpired queue whose creation time is closest to the expired queue, and determining the number of second log queues between the two, which is recorded as the number of intermediate queues; wherein the first log queue is the unexpired queue;

[0090] S15: The larger the number of the second log queues, the longer the delayed destruction time set for the expired queues;

[0091] S16: Determine whether the expired queue stores new logs within the delayed destruction time; if so, execute S17; if not, execute S18;

[0092] S17: Modify the expired queue to a non-expired queue;

[0093] S18: Destroy the expired queue.

[0094] Among them, the expiration time and delayed destruction time are both time ranges, and the calculation formula for the delayed destruction time is as follows:

[0095] T i =T A +T B *C;

[0096] Where i is the number of the second log queue, and the number is set in the order of creation; T i is the delayed destruction time of the expired queue numbered i; T A is the preset basic time, which is a fixed value; T B is a preset increase value, which is also a fixed value; C is the number of intermediate queues corresponding to the expired queue numbered i, specifically, the number of second log queues created between the expired queue numbered i and the unexpired queue closest to its creation time.

[0097] In S9, the network gate is provided with a first consumption thread, which writes the stored log into a preset file according to a preset strategy, referring to Figure 4 The flow chart of the preset strategy shown in the figure shows that the method for processing logs by the first consumer thread includes the following steps:

[0098] YY+242252P

[0099] S91: Determine whether the first log queue is empty; if not, execute S92; if yes, execute S93;

[0100] S92: The first consumer thread writes the logs in the first log queue into a preset file;

[0101] S93: Access the second log queue in the order of creation to determine whether there is a log in the second log queue and no corresponding second consumer thread; if so, execute S94; if not, execute S95;

[0102] S94: the first log queue writes the log in the accessed second log queue into a file;

[0103] S95: the count value is increased by 1, and the initial value of the count value is 0;

[0104] S96: Determine whether the count value reaches a preset value; if so, execute S97; if not, return to S91;

[0105] S97: The first log queue enters sleep mode and the sleep time is recorded;

[0106] S98: Determine whether the sleep time reaches the preset sleep threshold; if so, return to S97 and continue sleeping; if not, execute S99;

[0107] S99: The first log queue is awakened, the sleep time and the count value are reset to zero, and the process returns to S91.

[0108] Through the above preset strategy, the first log queue can get the maximum use value, thereby reducing the number of second consumer threads created, reducing system consumption, and improving the efficiency of thread management and scheduling. At the same time, when there are no new logs for a long time, the first log queue is put to sleep to save CPU resources and further reduce system consumption.

[0109] In S10, when the storage capacity of the second log queue reaches the capacity threshold, the current log receiving rate and the previous log receiving rate are obtained, wherein the current log receiving rate is equal to the number of logs received by the network gate in the current time step, and the previous log receiving rate is equal to the number of logs received by the network gate in the previous time step. The present disclosure sets 1 time step to 1 second. If the difference between the current log receiving rate and the previous log receiving rate is positive, it means that the current log receiving rate is increasing; if the difference is negative, it means that the current log receiving rate is decreasing; if the difference is 0, it means that the current log receiving rate is stable. If it is an increasing trend, create a second consumer thread to write the logs in the second log queue to the file

[0110] When the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate is on an increasing trend, a dedicated second consumption thread is created for the second log queue, and the second consumption thread is responsible for writing the logs in the second log queue to the file. If the storage capacity of the second log queue does not reach the capacity threshold, or the current log receiving rate is not on an increasing trend, there is no need to create a dedicated second consumption thread for the second log queue.

[0111] Furthermore, each second log queue has an independent rate threshold, which refers to the current log receiving rate saved when the second log queue is created. When there is no log in the second log queue with the second consumer thread and the current log receiving rate is less than the corresponding rate threshold, the second consumer thread is closed.

[0112] The efficiency of writing logs with a single thread is low and may cause blockage. Although the method of using multiple threads to write logs to the same file can improve the writing speed, it also brings about the problem of lock competition and frequent resource creation, which may cause system performance to decline. The method of writing logs to multiple files with multiple threads will generate too many files when the log receiving rate is low, resulting in a waste of server resources. The cross-network log processing method disclosed in the present invention can reduce the number of threads created when the amount of logs is small, while not affecting the log processing, saving system resources as much as possible. When the amount of logs is large, the second log queue and the number of threads created will also increase appropriately, while ensuring efficient log processing, avoiding the waste of system resources as much as possible.

[0113] In S11, in a specific implementation, the method for determining whether the file writing is completed includes: when the number of logs written to the file reaches a preset number threshold, or the writing time of the file reaches a preset time threshold, determining that the file writing is completed; when the number of logs written to the file does not reach the preset number threshold, and the writing time of the file does not reach the preset time threshold, determining that the file writing is not completed. Among them, the writing time of the file refers to the time when the file is written by the log, which can also be understood as the time when the file receives the log, and the number threshold can be set to 1000.

[0114] In another specific implementation, a method for determining whether file writing is completed includes: first determining whether the number of logs written to the file reaches a preset number threshold; if so, determining that the file writing is completed; if not, determining whether the file writing time reaches a preset time threshold; if so, determining that the file writing is completed; if not, the file writing is not completed.

[0115] Furthermore, the network gate is provided with a file queue and an upload thread group, the upload thread group includes several upload threads, when the file writing is completed, the path of the file is added to the file queue, the upload thread in the upload thread group continuously obtains the path from the file queue, finds the corresponding file according to the path and uploads the file to the second network, after the upload is completed, determines whether the file queue is empty; if so, the upload thread in the upload thread group waits; if not, continues the upload operation.

[0116] Note that the cross-network log processing method described above can try to write logs to files in the order in which they are received by the gateway, to ensure that many logs can be arranged in chronological order for easy management. When the file is written and the file queue is not empty, a new file is created and the newly received logs are written to the new file. Compared to the number of logs received, the number of files generated is much smaller, and usually only one file queue is used to implement system operation.

[0117] However, the number of upload threads in the upload thread group can be dynamically adjusted according to the number of files to save system resources while meeting the transmission efficiency. In one embodiment, the production rate and consumption rate of the file queue are obtained at each preset time interval. If the production rate is greater than the consumption rate, it means that the upload rate is low, and the current number of threads in the upload thread group is increased; if the production rate is less than the consumption rate, it means that the file consumption efficiency is high, and in order to avoid resource waste, the current number of threads in the upload thread group is reduced; if the production rate is equal to the consumption rate, it means that the file generation and consumption rates are relatively balanced, and the current number of threads in the upload thread group does not need to be adjusted.

[0118] Among them, the production rate is equal to the number of paths of files that have been written successfully added to the file queue in the current time step, and the consumption rate is equal to the number of files that the upload thread group successfully uploads to the second network through the file queue in the current time step.

[0119] In this embodiment, the processing status of the file can be determined in time through the current production rate and consumption rate, and the current number of threads of the upload thread group can be adjusted in time.

[0120] In another embodiment, a specific method for processing an upload thread group includes: obtaining the production rate and consumption rate of the file queue at each time step within a preset time interval. Obtain the difference between the production rate and the consumption rate at each time step, and count the number of positive and negative differences. If the number of positive differences is greater than the number of negative differences, it means that the upload rate is low, and the current number of threads of the upload thread group is increased; if the number of negative differences is greater than the number of positive differences, it means that the file consumption efficiency is high, and in order to avoid resource waste, the current number of threads of the upload thread group is reduced; if the number of positive differences is equal to the number of negative differences, it means that the file generation and consumption rates are relatively balanced, and the current number of threads of the upload thread group does not need to be adjusted.

[0121] In this embodiment, the production rate and consumption rate within a period of time can be collected, and the file processing status within the period of time can be evaluated through the difference, so as to make a more accurate adjustment of the current number of threads.

[0122] In yet another embodiment, referring to Figure 5 The flowchart of the dynamic adjustment of the upload thread group is shown, and the specific method of processing the upload thread group includes:

[0123] S111: obtaining the production rate and consumption rate of the file queue at each time step within a preset time interval;

[0124] S112: Based on the production rate and consumption rate, obtain the ideal number of threads required for each time step;

[0125] S113: When the time interval is reached, the current number of threads of the upload thread group is adjusted based on the ideal number of threads required for each time step.

[0126] For example, if the time interval is set to 2 minutes and the time step is 1 second, then the production rate and consumption rate are obtained 120 times every 2 minutes, and the ideal number of threads is calculated 120 times based on this. The current number of threads of the upload thread group is equal to the average of the ideal number of threads required for each time step. Therefore, the average of the ideal number of threads calculated 120 times is taken, and the current number of threads of the upload thread group is adjusted to be equal to the average.

[0127] The calculation formula for the ideal number of threads is:

[0128] L t =S t (X t / l t );

[0129] Where t is the time step at that time; L t is the ideal number of threads at the time step; S tis the production rate of the file queue at the current time step; X t is the consumption rate of the file queue at the current time step; l t The actual number of threads in the upload thread group at the current time step.

[0130] This embodiment calculates the ideal number of threads within a period of time, and then calculates the average value to find the current most ideal number of upload threads, which can adapt to the current file processing as much as possible.

[0131] Management scene.

[0132] The above contents all adopt the delayed adjustment method, that is, non-real-time adjustment, to avoid unstable fluctuations in the number of upload threads, which will cause a heavy burden on the system. By adjusting the number of upload threads at intervals, it not only avoids large fluctuations in the number of upload threads, but also tries to ensure the timeliness of the adjustment, so as to adjust the file processing efficiency to the best as soon as possible.

[0133] In summary, the cross-network log processing method of the present invention overcomes the problems of fluctuating log receiving rate, low efficiency of writing files, waste of server resources, decoupling of writing and uploading, etc. This method dynamically adjusts the use of server resources by monitoring the log receiving rate, reuses existing server resources to avoid resource waste, and uses queues to reduce the coupling between writing files and uploading files, thereby greatly improving the efficiency of writing files and uploading files and making rational use of server resources.

[0134] Reference Figure 6 The present disclosure provides a cross-network log processing system, including:

[0135] The log real-time receiving module 101 is used to receive logs from the first network in real time;

[0136] The log storage queue module 102 is used to store the log in the first resident log queue when the storage capacity of the first resident log queue meets the condition;

[0137] The log queue creation module 103 is used to create a temporary log queue second log queue to store the log when the storage capacity of the permanent log queue first log queue reaches the capacity threshold and there is no second log queue;

[0138] The first log writing module 104 is used to write the stored log into a preset file through the first resident consumer thread;

[0139] The second log writing module 105 is used to create a temporary consumption thread second consumption thread to write the logs in the temporary log queue second log queue into the file when the storage capacity of the temporary log queue second log queue reaches the capacity threshold and the current log receiving rate shows an increasing trend;

[0140] The file upload network module 106 is used to upload the file to the second network after the file is written.

[0141] The various variations and specific examples of the cross-network log processing method provided above are also applicable to the cross-network log processing system provided in the present disclosure. Through the above detailed description of the cross-network log processing method, those skilled in the art can clearly know the implementation method of the cross-network log processing system. For the sake of brevity of the specification, it will not be described in detail here.

[0142] The computer device according to the embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0143] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the computer device executes all or part of the steps of the cross-network log processing method of each embodiment of the present disclosure.

[0144] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0145] like Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure is shown, which is a schematic diagram of the structure of a computer device suitable for implementing the embodiment of the present disclosure. Figure 7 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0146] like Figure 7As shown, the computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0147] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 7 A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0148] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the cross-network log processing method of an embodiment of the present disclosure are executed.

[0149] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0150] According to the computer-readable storage medium of the embodiment of the present disclosure, non-transitory computer-readable instructions are stored thereon. When the non-transitory computer-readable instructions are executed by the processor, all or part of the steps of the cross-network log processing method of each embodiment of the present disclosure are executed.

[0151] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0152] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0153] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0154] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0155] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0156] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0157] Various changes, substitutions, and alterations to the techniques described herein may be made without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. There may be currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein. Thus, the appended claims are included in YY+242252P

[0158] Such process, machine, manufacture, composition of matter, means, methods, or actions within its scope.

[0159] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0160] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A cross-network log processing method, characterized in that: include: Receive logs from the first network in real time; When the storage capacity of the first log queue meets the condition, storing the log in the first log queue; When the storage capacity of the first log queue reaches a capacity threshold and there is no second log queue, creating a second log queue to store the log; The second log queue writes the stored logs to a preset file through the first consumer thread; When the storage capacity of the second log queue reaches the capacity threshold and the current log receiving rate shows an increasing trend, a second consumer thread is created to write the logs in the second log queue into the file; After the file is written, the file is uploaded to the second network.

2. The cross-network log processing method according to claim 1, characterized in that: When the storage capacity of the first log queue and the existing second log file both reach the capacity threshold, a new second log queue is created to store logs.

3. The cross-network log processing method according to claim 1, characterized in that: When the second log queue does not store new logs within a preset expiration time, the second log queue is delayedly destroyed.

4. The cross-network log processing method according to claim 3, characterized in that: Writing the stored log into a preset file through the first consumer thread includes: The first consumer thread preferentially writes the logs in the first log queue into the file; YY+242252P If the first log queue is empty, the second log queue is accessed in the order of creation. If the accessed second log queue contains logs but has no corresponding second consumer thread, the first log queue writes the logs in the accessed second log queue into a file.

5. The cross-network log processing method according to claim 1, characterized in that: Each secondary log queue has a rate threshold; When there is no log in the second log queue having the second consumer thread, and the current log receiving rate is less than the corresponding rate threshold, the second consumer thread is closed.

6. The cross-network log processing method according to claim 1, characterized in that: When the number of logs written into the file reaches a preset number threshold, or the writing time of the file reaches a preset time threshold, it is determined that the writing of the file is completed.

7. The cross-network log processing method according to claim 1, characterized in that: When the file writing is completed and the file queue is not empty, a new file is created for writing the log.

8. The cross-network log processing method according to claim 1, characterized in that: After the file is written, uploading the file to the second network includes: Uploading the file to the second network through a preset file queue and upload thread group; The first log queue is a permanent log queue, and the second log queue is a temporary log queue; the first consumer thread is a permanent consumer thread, and the second consumer thread is a temporary consumer thread.

9. The cross-network log processing method according to claim 8, characterized in that: The uploading of the file to the second network through a preset file queue and an upload thread group includes: In the preset time interval, obtain the production speed of the file queue at each time step YY+242252P rate and consumption rate; Based on the production rate and the consumption rate, obtain the ideal number of threads required for each time step; When the time interval is reached, adjusting the current number of threads of the upload thread group based on the ideal number of threads required for each time step; Adding the path of the written file to the file queue, and the upload thread group uploading the corresponding file to the second network based on the path; The calculation formula for the ideal number of threads is: L t =S t (X t / l t ); Among them, t is the time step at that time; L t is the ideal number of threads at the time step; S t is the production rate of the file queue at the current time step; X t is the consumption rate of the file queue at the current time step; l t The actual number of threads in the upload thread group at the current time step.

10. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the cross-network log processing method described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the cross-network log processing method described in any one of claims 1-9.

12. A computer program product comprising computer instructions, characterized in that: The plan is YY+242252P When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

Images