Log transmission method and device, electronic equipment and storage medium

Abstract

The invention provides a log transmission method and device, electronic equipment and a storage medium, relates to the technical field of computers, and can adjust the transmission configuration of each network communication mode into a target transmission configuration matched with an overall pressure value according to the overall pressure value of a local machine to prevent each network communication mode from excessively occupying hardware resources of the local machine. Besides, the hardware resource allocation proportion of each network communication mode can be determined according to the usage amount of each hardware resource, the type of the hardware resource on which each network communication mode depends and the transmission priority of the log data needing to be transmitted by each network communication mode, so that various network communication modes can be ensured to obtain appropriate hardware resources; and mutual resource scrambling is avoided. And finally, according to the target transmission configuration and the hardware resource allocation amount of each network communication mode, each network communication mode is adopted to transmit the log data in parallel, so that the log transmission performance can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log transmission method, device, electronic device, and storage medium. Background Art

[0002] To improve the log collection performance of server clusters, different network communication methods can be used for log transmission within log transmission nodes. However, these different network communication methods lack coordinated optimization of hardware resource usage, which can easily lead to insufficient hardware resources and competition for resources among different network communication methods, impacting log transmission performance. Summary of the Invention

[0003] The present application provides a log transmission method, device, electronic device and storage medium. On the one hand, the transmission configuration of each network transmission mode can be adjusted according to the overall pressure value of the local machine to avoid excessive use of the local machine's hardware resources. On the other hand, an appropriate amount of hardware resources can be allocated to each network transmission mode to avoid resource competition between network transmission modes, thereby improving log transmission performance.

[0004] This application provides a log transmission method, including:

[0005] Obtaining log data, determining the log data to be transmitted by each network communication mode in at least two network communication modes, and determining the transmission priority of each log data;

[0006] Obtaining hardware resource usage of at least two types of hardware resources, determining an overall pressure value of the local device based on the hardware resource usage, and selecting a target transmission configuration that matches the overall pressure value from at least two preset transmission configurations corresponding to each network communication mode;

[0007] Determine the hardware resource allocation ratio for each network communication method based on the usage of each hardware resource, the type of hardware resources that each network communication method relies on, and the transmission priority of the log data required to be transmitted by each network communication method; and determine the hardware resource allocation amount for each network communication method based on the hardware resource amount of each hardware resource type and the hardware resource allocation ratio;

[0008] Based on the target transmission configuration and hardware resource allocation of each network communication method, log data is transmitted in parallel using each network communication method.

[0009] The present invention also provides a log transmission device, comprising:

[0010] a log processing module, configured to obtain log data, determine the log data to be transmitted by each network communication mode in at least two network communication modes, and determine the transmission priority of each log data;

[0011] a configuration adjustment module, configured to obtain hardware resource usage of at least two types of hardware resources, determine an overall pressure value of the local machine based on the hardware resource usage, and select a target transmission configuration that matches the overall pressure value from at least two preset transmission configurations corresponding to each network communication mode;

[0012] a resource allocation module for determining the hardware resource allocation ratio for each network communication method based on the usage of each hardware resource, the type of hardware resources that each network communication method relies on, and the transmission priority of the log data required to be transmitted by each network communication method, and determining the hardware resource allocation amount for each network communication method based on the hardware resource amount of each hardware resource type and the hardware resource allocation ratio;

[0013] The log transmission module is used to transmit log data in parallel using various network communication methods according to the target transmission configuration and hardware resource allocation of each network communication method.

[0014] The present invention further provides an electronic device, comprising:

[0015] memory for storing computer programs;

[0016] The processor is used to implement the above-mentioned log transmission method when executing a computer program.

[0017] The present invention also provides a non-volatile computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the above-mentioned log transmission method is implemented.

[0018] The beneficial effects of the present invention are as follows: the present invention can first obtain log data, determine the log data required to be transmitted by each network communication method in at least two network communication methods, and determine the transmission priority of each log data. Subsequently, the present invention can also obtain the hardware resource usage of at least two hardware resource types, determine the overall pressure value of the machine based on the hardware resource usage, and select the target transmission configuration that matches the overall pressure value from the at least two preset transmission configurations corresponding to each network communication method, that is, ensure that the transmission configuration of each network communication method meets the current pressure of the machine and avoid hardware resource shortages. Subsequently, the present invention can also determine the hardware resource allocation ratio of each network communication method based on the hardware resource usage of each hardware resource type, the hardware resource type that each network communication method relies on, and the transmission priority of the log data required to be transmitted by each network communication method, and determine the hardware resource allocation amount of each network communication method based on the hardware resource amount of each hardware resource type and the hardware resource allocation ratio, thereby ensuring that each network communication method can obtain appropriate hardware resources and avoid competing for resources. Finally, based on the target transmission configuration and hardware resource allocation amount of each network communication method, log data can be transmitted in parallel by each network communication method, which can effectively improve log transmission performance.

[0019] The present invention also provides a log transmission device, electronic equipment and storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A flow chart of a log transmission method provided by an embodiment of the present invention;

[0022] Figure 2 A flow chart of another log transmission method provided by an embodiment of the present invention;

[0023] Figure 3 A flowchart of a thread adjustment process provided by an embodiment of the present invention;

[0024] Figure 4 A structural block diagram of a log transmission device provided by an embodiment of the present invention;

[0025] Figure 5 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0028] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] To improve the log collection performance of server clusters, different network communication methods can be used within log transmission nodes. For example, TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) can be used for direct log data transmission, or message-based middleware (such as Kafka) can be used for log data transmission. However, different network communication methods lack coordinated optimization of hardware resource usage, which can easily lead to insufficient hardware resources and competition for resources among different network communication methods, impacting log transmission performance.

[0030] In view of this, the present invention can provide a log transmission method, which can, on the one hand, adjust the transmission configuration of each network transmission mode according to the overall pressure value of the local machine to avoid excessive use of the local machine's hardware resources; on the other hand, it can allocate an appropriate amount of hardware resources to each network transmission mode to avoid resource competition between network transmission modes, thereby improving log transmission performance.

[0031] For easier understanding, please refer to Figure 1 , Figure 1 A flow chart of a log transmission method provided by an embodiment of the present invention may include:

[0032] S101: Obtain log data, determine the log data to be transmitted in each network communication mode among at least two network communication modes, and determine the transmission priority of each log data.

[0033] In this embodiment, the log transmission node can use at least two network communication methods to transmit log data in parallel. Therefore, when acquiring log data, this step first needs to determine the log data to be transmitted by each network communication method. For example, the network communication method used to transmit the log data can be determined by the port number or message header (such as the headers field in a Kafka message) used to transmit the log data.

[0034] Furthermore, to ensure the transmission of important log data, this step also requires determining the priority of each log data item. For example, you can set the transmission priority based on key fields in the log content. For example, log data containing the [ERROR] field can be assigned a high transmission priority, log data containing the [WARN] field can be assigned a medium transmission priority, and log data containing the [Debug] field can be assigned a low transmission priority. This prioritizes the transmission of log data containing error and warning content. Furthermore, log data can be determined based on its data source, prioritizing the transmission of log data from key data sources.

[0035] Based on this, determine the transmission priority of log data, which may include:

[0036] Step 11: extract the fields to be detected from the log data, match the fields to be detected with the preset fields of each transmission priority, and determine the transmission priority of the log data according to the matching result.

[0037] In addition, determining the transmission priority of log data may also include:

[0038] Step 21: Determine the transmission priority of the log data according to the data source of the log data.

[0039] Furthermore, considering that log data may be difficult to transmit in a timely manner, a cache structure can be provided to store log data. This embodiment does not limit the specific cache structure; for example, a queue, a ring buffer, etc. can be used. Considering that a ring buffer facilitates data writing and reading, a ring buffer can be used to store log data.

[0040] In addition, it is also considered that the log data may be squeezed, thereby causing the annular buffer area to be full. Therefore, when it is determined that the annular buffer area is full, this embodiment can also store the log data in a memory (such as an SSD, solid state memory).

[0041] Based on this, the method may further include:

[0042] Step 31: Determine whether the ring buffer is full;

[0043] Step 32: If the ring buffer is not full, add the log data to the ring buffer;

[0044] Step 33: If the ring buffer is full, the log data is added to the memory.

[0045] S102. Obtain hardware resource usage of at least two types of hardware resources, determine an overall pressure value of the local device based on the hardware resource usage, and select a target transmission configuration that matches the overall pressure value from at least two preset transmission configurations corresponding to each network communication mode.

[0046] In this embodiment, considering that inappropriate transmission configuration (such as transmission rate, cache size, cache time, transmission algorithm) can easily lead to excessive use of hardware resources by each network communication method, such as excessive use of processor resources, network bandwidth resources, memory resources, disk IO resources, etc., this step will first adjust the transmission configuration of each network communication method based on the overall pressure of the local machine to constrain the use of local hardware resources by each network communication method. To this end, this step can first obtain the hardware resource usage of at least two types of hardware resources, such as processor usage, network bandwidth usage, memory usage, disk IO bandwidth usage, etc., and can determine the overall pressure value of the local machine based on the hardware resource usage. Subsequently, this step can select a target transmission configuration that matches the overall pressure value from at least two preset transmission configurations corresponding to each network communication method to ensure that each network communication method can reasonably use hardware resources.

[0047] It should be noted that this embodiment does not limit how to determine the overall pressure value. For example, the usage of various hardware resources can be simply weighted. However, considering that simple weighting may be difficult to reflect the scarcity of some hardware resources, such as for some hardware resources with lower weights, even if their usage is large and resources are relatively scarce, it may be difficult to effectively increase the overall pressure value. Therefore, before performing weighted summation on the hardware resource usage, this embodiment may first obtain the historical hardware resource usage of each hardware resource to determine whether the hardware resource has also had a large usage and scarce resources in the past historical moments. If so, this embodiment may increase the initial weight of the hardware resource to highlight the scarcity of the hardware resource in the overall pressure value. In this way, this embodiment can ensure that the overall pressure value better reflects the pressure situation of the machine.

[0048] Based on this, the overall stress value of the machine is determined based on the hardware resource usage, which may include:

[0049] Step 41: Obtain preset initial weights of various hardware resource types, and obtain at least two historical hardware resource usages corresponding to the various hardware resource types.

[0050] It should be noted that this embodiment does not limit the preset initial weights of each hardware resource type, which can be set according to actual application requirements. This embodiment also does not limit how much historical hardware resource usage needs to be obtained for each hardware resource type, which can be set according to actual application requirements.

[0051] Step 42: Determine whether the historical hardware resource usage of each hardware resource type is greater than a preset threshold.

[0052] Step 43: If the historical hardware resource usage of the hardware resource type is greater than the preset threshold, the preset initial weight of the hardware resource type is increased to obtain the weighted weight of the hardware resource type.

[0053] Step 44: If the historical hardware resource usage of the hardware resource type is not greater than the preset threshold, the preset initial weight of the hardware resource type is used as the weighted weight of the hardware resource type.

[0054] In steps 42-44, if the historical hardware resource usage of a certain hardware resource type is consistently greater than the preset threshold, this indicates that the hardware resource type has been in a state of high usage and resource scarcity for a long time, and its preset initial weight needs to be increased to highlight its scarcity. If the historical hardware resource usage of a certain hardware resource type is not consistently greater than the preset threshold, this indicates that the hardware resource type has not been in a state of high usage and resource scarcity for a long time, and its initial weight can be maintained unchanged.

[0055] Step 45: Perform weighted processing on the usage of each hardware resource according to the weighted weight of each hardware resource type to obtain an overall pressure value.

[0056] Furthermore, considering that log transmission requires a lot of network resources, when it is determined that the local machine is under high pressure for a long time, in order to save network resources as much as possible, the preset initial weight of network resources can be increased. Based on this, the method can also include:

[0057] Step 51: Obtain at least two historical overall pressure values;

[0058] Step 52: Determine whether all historical overall pressure values ​​are greater than a preset threshold;

[0059] Step 53: If the historical overall pressure values ​​are all greater than the preset threshold, the preset initial weight of the network resource in the hardware resource category is increased to obtain the weighted weight of the network resource;

[0060] Step 54: If the historical overall pressure values ​​are not all greater than the preset threshold, the preset initial weight of the network resource is used as the weighted weight of the network resource.

[0061] In steps 51 to 54, at least two historical overall pressure values ​​of the local machine can be obtained, such as at least two recent historical overall pressure values. Subsequently, it can be determined whether the historical overall pressure values ​​are all greater than a preset threshold. If the historical overall pressure values ​​are all greater than the preset threshold, it means that the local machine has been in a high-pressure state for a long time, and the preset initial weight of the network resources in the hardware resource category needs to be increased to highlight the scarcity of network resources, thereby adjusting the transmission configuration of each network transmission mode according to the current overall pressure value to save network resources. Conversely, if the historical overall pressure values ​​are not all greater than the preset threshold, the preset initial weight of the network resources can be kept unchanged.

[0062] The following is a specific example to introduce the calculation method of the overall pressure value. The overall pressure value can be expressed as:

[0063] Overall pressure value = α×CPU utilization + β×network retransmission rate + γ× + δ×memory shortage index;

[0064] Among them, α, β, γ, and δ are weights, and the initial weights are: α=0.5, β=0.2, γ=0.2, δ=0.1.

[0065] The following are the definitions and calculations of CPU utilization, network retransmission rate, swap space usage, and memory shortage index:

[0066] CPU utilization: can be calculated from the records in / proc / stat using the following formula:

[0067] CPU utilization (%) = [(user state time + kernel state time) / total time slice] × 100;

[0068] Among them, user-mode time refers to the time consumed by the CPU executing application code in user space, while kernel-mode time refers to the time consumed by the CPU executing operating system code in kernel space.

[0069] Network retransmission rate: You can obtain the number of TCP retransmitted packets (TCPRetransSegs) and the total number of sent packets (TCPOutSegs) through / proc / net / snmp and calculate it using the following formula:

[0070] Network retransmission rate (%) = (TCPRetransSegs / TCPOutSegs) × 100;

[0071] When it is determined that the packet loss rate (such as the UDP packet loss rate) is greater than a preset value, the network retransmission rate may be weighted by ×1.2 to improve network sensitivity.

[0072] Swap space usage: You can obtain the total amount of swap (SwapTotal) and the remaining amount (SwapFree) from / proc / meminfo and calculate it using the following formula:

[0073] Swap usage (%) = [(SwapTotal - SwapFree) / SwapTotal] × 100;

[0074] Note that in this embodiment, the Swap usage rate can be squared to amplify the impact of high Swap usage rates (for example, when the Swap usage rate is 80%, the contribution value = 0.2 × 80² = 1280, while the linear term is only 16).

[0075] Memory shortage index: The memory shortage index can be a Boolean variable, which is set to 1 when the free memory (MemFree) is less than 10% of the total memory, otherwise it is 0.

[0076] It is worth noting that this embodiment allows adjustments to the aforementioned weights. For example, if the stress index is > 75 for three consecutive times, the network weight β is increased to 0.3 to accommodate network-intensive scenarios. If the memory shortage index remains at 1 for more than five minutes, the γ is increased to 0.3 to enhance the memory pressure response.

[0077] Furthermore, to adjust the transmission configuration corresponding to each network communication method based on the overall pressure value, at least two preset overall pressure value intervals can be preset, and a preset transmission configuration corresponding to each preset overall pressure value interval can be set for each network communication method. Consequently, it is only necessary to determine the target overall pressure value interval within which the overall pressure value falls and adjust the transmission configuration of the network communication method to the configuration corresponding to the target overall pressure value interval.

[0078] Based on this, a target transmission configuration that matches the overall pressure value is selected from at least two preset transmission configurations corresponding to each network communication mode, including:

[0079] Step 61: determining a target overall pressure value interval in which the overall pressure value lies in at least two preset overall pressure value intervals;

[0080] Step 62: Determine, among the preset transmission configurations corresponding to the network communication mode, a preset transmission configuration that matches the target overall pressure value interval as the target transmission configuration.

[0081] In one embodiment, the preset overall pressure value intervals can be divided into 0-40, 40-60, 60-80, and 80-maximum value. The transmission rate and transmission strategy corresponding to each interval are as follows:

[0082] Table 1 Transmission configuration adjustment diagram

[0083]

[0084] S103. Determine the hardware resource allocation ratio of each network communication method based on the usage of each hardware resource, the type of hardware resources that each network communication method depends on, and the transmission priority of the log data required to be transmitted by each network communication method, and determine the hardware resource allocation amount of each network communication method based on the hardware resource amount of each hardware resource type and the hardware resource allocation ratio.

[0085] In this step, after adjusting the transmission configuration for each network communication method, a reasonable amount of hardware resources can be allocated to each network communication method. First, this embodiment can set an initial weight for each network communication method to determine the hardware resource allocation ratio. Furthermore, it is understandable that different network communication methods use different hardware resources. For example, TCP consumes more CPU and network resources. UDP, because it does not require ACK messages, only consumes more network bandwidth. When using message middleware to transmit log data, disk I / O is consumed more. In this case, because a specific network communication method consumes more of the specific hardware resources it relies on, it can easily affect the use of these hardware resources by other network communication methods. Therefore, it is necessary to limit the use of the specific hardware resources that a specific network communication method relies on. To this end, this embodiment can determine whether various hardware resources are in short supply based on the usage of each hardware resource. If a hardware resource shortage is determined, the network transmission method that relies on the hardware resource can be searched and its initial weight can be adjusted downward to limit its use of the hardware resource.

[0086] Furthermore, since the present embodiment has previously set a transmission priority for each log data, it is understandable that log data with a high transmission priority should be given priority to be provided with hardware resources for data transmission. Therefore, after adjusting the initial weight of each network transmission method, the urgency correction weight of each network communication method can also be determined according to the transmission priority of the log data transmitted by each network communication method, and the initial weight of the network communication method can be further corrected using the urgency correction weight. Among them, the weight value of the urgency correction weight is proportional to the transmission priority, that is, the higher the transmission priority, the greater the urgency correction weight, and the higher the resources allocated to the corresponding network transmission method. In this way, the present embodiment can ensure that each network transmission method has sufficient resources to use, avoiding competition for resources between different network transmission methods. At the same time, it can also ensure that high-priority log data can obtain sufficient resources for transmission, thereby improving the reliability of resource allocation.

[0087] Based on this, the hardware resource allocation ratio for each network communication method is determined based on the hardware resource usage, the type of hardware resources each network communication method relies on, and the transmission priority of the log data transmitted by each network communication method, including:

[0088] Step 71: Obtaining the preset initial weights of each network communication mode;

[0089] Step 72: For each hardware resource type, determine whether the hardware resource usage of the hardware resource type is greater than a preset threshold;

[0090] Step 73: If the hardware resource usage of the hardware resource type is not greater than the preset threshold, then for the network communication mode that depends on the hardware resource type, the preset initial weight of the network communication mode is used as the basic weight of the network communication mode;

[0091] Step 74: If the hardware resource usage of the hardware resource type is greater than the preset threshold, a preset resource pressure coefficient is obtained, and for the network communication mode that depends on the hardware resource type, the preset initial weight of the network communication mode is adjusted downward using the preset resource pressure coefficient to obtain a basic weight of the network communication mode; the value of the preset resource pressure coefficient is between 0 and 1;

[0092] Step 75: Determine the urgency correction weight of each network communication mode according to the transmission priority of the log data transmitted by each network communication mode; the weight value of the urgency correction weight is proportional to the transmission priority;

[0093] Step 76: summing the basic weight of the network communication mode and the urgency correction weight to obtain the dynamic weight of the network communication mode, and determining the hardware resource allocation ratio of each network communication mode according to the dynamic driving of each network communication mode.

[0094] The following describes the process of setting hardware resources for each network communication mode based on specific examples. In one embodiment, the calculation formula of the dynamic weight can be:

[0095] Dynamic weight = basic weight × resource pressure coefficient + urgency correction;

[0096] The base weight is the default initial weight for each network transmission method. Regarding the resource pressure coefficient, when the pressure on a resource (such as the CPU) exceeds 80%, the weight of the protocols that rely on that resource is reduced (for example, TCP weight × 0.8). For urgency correction, it can be adjusted based on the log level (ERROR level +0.2, WARN level +0.1, DEBUG level -0.1).

[0097] This embodiment can perform dynamic resource allocation according to the above dynamic weights, specifically as follows:

[0098] Resource allocation formula:

[0099] Communication method Resource quota = total resources × ( / Σdynamic weight);

[0100] The total resources are the upper limit of the currently available resources (e.g., bandwidth = 100Mbps, CPU = 100%). is the dynamic weight of the communication mode in the i-th order.

[0101] Allocation example:

[0102] Available bandwidth = 100 Mbps, protocol weights: Kafka = 0.9, TCP = 0.2, UDP = 0.15;

[0103] Total weight = 0.9 + 0.2 + 0.15 = 1.25;

[0104] Kafka bandwidth quota = 100 Mbps × (0.9 / 1.25) = 72 Mbps;

[0105] TCP bandwidth quota = 100 × (0.2 / 1.25) = 16 Mbps;

[0106] UDP bandwidth quota = 100 × (0.15 / 1.25) = 12 Mbps.

[0107] S104 , using each network communication mode to transmit log data in parallel according to the target transmission configuration and hardware resource allocation of each network communication mode.

[0108] In this step, after completing the target transmission configuration and hardware resource allocation settings, each network communication method can be used to transmit log data in parallel according to the target transmission configuration and hardware resource allocation of each network communication method.

[0109] Specifically, the log data may be read from a cache structure (such as a ring cache or a memory) and transmitted.

[0110] Based on this, various network communication methods are adopted to transmit log data in parallel, including:

[0111] Step 81: Read log data from the ring buffer or memory, and transmit the log data in parallel using various network communication modes.

[0112] Finally, to facilitate understanding of the complete processing flow of this embodiment, please refer to Figure 2 , Figure 2 This is a flowchart of another log transmission method provided by an embodiment of the present invention.

[0113] Based on the above embodiments, the present invention can first obtain log data, determine the log data required to be transmitted by each network communication method in at least two network communication methods, and determine the transmission priority of each log data. Subsequently, the present invention can also obtain the hardware resource usage of at least two hardware resource types, determine the overall pressure value of the machine based on the hardware resource usage, and select the target transmission configuration that matches the overall pressure value from the at least two preset transmission configurations corresponding to each network communication method, that is, ensure that the transmission configuration of each network communication method meets the current pressure of the machine and avoid hardware resource shortages. Subsequently, the present invention can also determine the hardware resource allocation ratio of each network communication method based on the hardware resource usage of each hardware resource type, the hardware resource type that each network communication method relies on, and the transmission priority of the log data required to be transmitted by each network communication method, and determine the hardware resource allocation amount of each network communication method based on the hardware resource amount of each hardware resource type and the hardware resource allocation ratio, so as to ensure that each network communication method can obtain appropriate hardware resources and avoid mutual competition for resources. Finally, based on the target transmission configuration and hardware resource allocation amount of each network communication method, log data can be transmitted in parallel by each network communication method, which can effectively improve log transmission performance.

[0114] Based on the above embodiment, when multiple overall pressure value intervals are set, in order to ensure that the current thread of the local machine can stably process log transmission and reduce the fluctuation of the overall pressure value, an overall pressure target value can also be set for each overall pressure value interval, and the number of threads can be adjusted based on the overall pressure target value as the pressure value target, so that the overall pressure value of the local machine tends to the overall pressure target value. The following describes the process of setting the number of threads. Based on this, the overall pressure value is periodically detected; this method also includes:

[0115] S201: Obtain an overall pressure target value corresponding to a target overall pressure value interval.

[0116] It should be noted that each overall pressure range can have an overall pressure target value set. This way, when the overall pressure value of the machine fluctuates within a certain overall pressure range, the overall pressure target value of that overall pressure range can be set as the target to adjust the number of threads, ensuring that the overall pressure value of the machine approaches the overall pressure target value, thereby improving the stability of the machine.

[0117] It should be noted that this embodiment does not limit the overall pressure target value for each interval and can be adjusted according to needs. For example, for the four intervals of 0-40, 40-60, 60-80, and 80-maximum, the corresponding overall pressure target values ​​can be 20, 40, 60, and 80.

[0118] S202 : Calculate the difference between the overall pressure target value and the overall pressure value to obtain a first control value at the current detection moment.

[0119] S203: Obtain first control values ​​at multiple historical detection moments, sum the first control value at the current detection moment with the first control values ​​at each historical detection moment, and multiply the sum by the detection cycle duration to obtain a second control value;

[0120] S204, determining the difference between the first control value at the current detection time and the first control value at the most recent historical detection time, and dividing the difference result by the detection cycle length to obtain a third control value;

[0121] S205 , weighting the first control value, the second control value, and the third control value to obtain a thread number adjustment value;

[0122] S206: Adjust the current number of threads of the local machine according to the thread number adjustment value.

[0123] To facilitate understanding of steps S202 to S205, the following is an introduction based on a specific thread number adjustment formula. In one embodiment, the thread number adjustment formula can be:

[0124] ΔNumber of threads = Kp×e(t) + Ki×∫e(t)dt + Kd×de(t) / dt;

[0125] in:

[0126] e(t) represents the first control value, e(t) = overall pressure target value (default 60) - current overall pressure value;

[0127] ∫e(t)dt represents the second control value, where ∫e(t)dt = Σ(e(t) × Δt), where Δt is the detection period (1 second). ∫e(t)dt is used to eliminate steady-state errors and accumulate historical deviations. The integration period of ∫e(t)dt can be adjusted as needed, for example, to 1 minute. Furthermore, if the pressure index exceeds the target value for 10 consecutive seconds, the second control value can be reset to zero to prevent excessive accumulation.

[0128] de(t) / dt represents the third control value, de(t) / dt = [e(t) - e(t-1)] / Δt. de(t) / dt is used to suppress overshoot and predict future error trends. A sliding average (with a 3-second window) can be performed on de(t) / dt to mitigate noise interference.

[0129] Kp, Ki, and Kd are the weights of the first, second, and third control values. Kp responds quickly to the current error; a larger value results in more aggressive adjustments. A possible value for Kp is 2.0 (empirical value), indicating that for every 1% deviation of the pressure index from the target value, the number of threads is adjusted by 2 units.

[0130] After obtaining the thread number adjustment value, the final number of threads can be:

[0131] New number of threads = current number of threads + Δ number of threads (limit range: minimum 10 threads, maximum 200 threads).

[0132] The following describes the above calculation process based on specific numerical values:

[0133] 1. Current pressure index = 70 → first control item e(t) = 60-70 = -10;

[0134] 2. Second control term = accumulated error over the past 3 seconds: -8 (t-1), -5 (t-2), -3 (t-3) → ∫e(t)dt = -16;

[0135] 3. The third control term = the current error change rate: (-10) - (-8) = -2 → de(t) / dt = -2;

[0136] 4. Adjustment amount = 2.0×(-10) + 0.5×(-16) + 1.2×(-2) = -20 -8 -2.4 = -30.4;

[0137] 5. New number of threads = current number of threads (200) + (-30.4) ≈ 170 (rounded down).

[0138] To understand the thread adjustment process of this embodiment, please refer to Figure 3 , Figure 3 The present invention provides a flowchart of a thread adjustment process.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0140] Please refer to Figure 4 , Figure 4 This is a structural block diagram of a log transmission device provided by an embodiment of the present invention. The device may include:

[0141] The log processing module 401 is used to obtain log data, determine the log data to be transmitted by each network communication mode in at least two network communication modes, and determine the transmission priority of each log data;

[0142] Configuration adjustment module 402 is configured to obtain hardware resource usage of at least two hardware resource types, determine an overall pressure value of the local device based on the hardware resource usage, and select a target transmission configuration that matches the overall pressure value from at least two preset transmission configurations corresponding to each network communication mode;

[0143] Resource allocation module 403, configured to determine the hardware resource allocation ratio for each network communication method based on the hardware resource usage of each hardware resource, the type of hardware resource that each network communication method relies on, and the transmission priority of log data required to be transmitted by each network communication method, and to determine the hardware resource allocation amount for each network communication method based on the hardware resource amount of each hardware resource type and the hardware resource allocation ratio;

[0144] The log transmission module 404 is used to transmit log data in parallel using each network communication mode according to the target transmission configuration and hardware resource allocation of each network communication mode.

[0145] Optionally, the configuration adjustment module 402 includes:

[0146] A first acquisition submodule is configured to acquire preset initial weights of various hardware resource types and to acquire at least two historical hardware resource usages corresponding to the various hardware resource types;

[0147] The first weight adjustment submodule is used to determine whether the historical hardware resource usage of the hardware resource types is greater than a preset threshold; if the historical hardware resource usage of the hardware resource types is greater than the preset threshold, the preset initial weight of the hardware resource types is adjusted upward to obtain a weighted weight of the hardware resource types; if the historical hardware resource usage of the hardware resource types is not greater than the preset threshold, the preset initial weight of the hardware resource types is used as the weighted weight of the hardware resource types;

[0148] The first weighting submodule is used to perform weighted processing on the usage of each hardware resource according to the weighted weight of each hardware resource type to obtain an overall pressure value.

[0149] Optionally, the configuration adjustment module 402 includes:

[0150] A second acquisition submodule is used to obtain at least two historical overall pressure values;

[0151] The second weight adjustment submodule is used to determine whether the historical overall pressure values ​​are all greater than the preset threshold; if the historical overall pressure values ​​are all greater than the preset threshold, the preset initial weight of the network resources in the hardware resource type is adjusted upward to obtain the weighted weight of the network resources; if the historical overall pressure values ​​are not all greater than the preset threshold, the preset initial weight of the network resources is used as the weighted weight of the network resources.

[0152] Optionally, the configuration adjustment module 402 includes:

[0153] an interval confirmation submodule, configured to determine a target overall pressure value interval in which the overall pressure value is located, from among at least two preset overall pressure value intervals;

[0154] The configuration adjustment submodule is used to determine, from the preset transmission configurations corresponding to the network communication mode, a preset transmission configuration that matches the target overall pressure value interval as the target transmission configuration.

[0155] Optionally, the device may further include:

[0156] A target value acquisition module is used to obtain the overall pressure target value corresponding to the target overall pressure value interval;

[0157] A first control value calculation module is used to calculate the difference between the overall pressure target value and the overall pressure value to obtain a first control value at the current detection moment;

[0158] a second control value calculation module, configured to obtain first control values ​​at multiple historical detection moments, sum the first control value at the current detection moment with the first control values ​​at each historical detection moment, and multiply the sum by the detection cycle length to obtain a second control value;

[0159] a third control value calculation module, configured to determine the difference between the first control value at the current detection moment and the first control value at the most recent historical detection moment, and divide the difference result by the detection cycle length to obtain a third control value;

[0160] a thread number adjustment value calculation module, configured to weight the first control value, the second control value, and the third control value to obtain a thread number adjustment value;

[0161] The thread adjustment module is used to adjust the current number of threads of the local machine according to the thread number adjustment value.

[0162] Optionally, the log processing module 401 includes:

[0163] The priority determination submodule is used to extract the fields to be detected from the log data, match the fields to be detected with the preset fields of each transmission priority, and determine the transmission priority of the log data based on the matching results; and / or determine the transmission priority of the log data based on the data source of the log data.

[0164] Optionally, the device may further include:

[0165] The cache module is used to determine whether the circular buffer area is full; if the circular buffer area is not full, the log data is added to the circular buffer area; if the circular buffer area is full, the log data is added to the memory;

[0166] The log transmission module 404 may include:

[0167] The reading submodule is used to read log data from the circular buffer area or memory and transmit the log data in parallel using various network communication methods.

[0168] Optionally, the resource allocation module 403 may include:

[0169] The initial weight acquisition submodule is used to obtain the preset initial weights of each network communication mode;

[0170] The weight adjustment submodule is used to determine, for each hardware resource type, whether the hardware resource usage of the hardware resource type is greater than a preset threshold; if the hardware resource usage of the hardware resource type is not greater than the preset threshold, then, for the network communication method that depends on the hardware resource type, the preset initial weight of the network communication method is used as the basic weight of the network communication method; if the hardware resource usage of the hardware resource type is greater than the preset threshold, then obtain a preset resource pressure coefficient, and for the network communication method that depends on the hardware resource type, use the preset resource pressure coefficient to lower the preset initial weight of the network communication method to obtain the basic weight of the network communication method; the value of the preset resource pressure coefficient is between 0 and 1;

[0171] The correction submodule is used to determine the urgency correction weight of each network communication mode according to the transmission priority of the log data transmitted by each network communication mode; the weight value of the urgency correction weight is proportional to the transmission priority;

[0172] The ratio determination submodule is used to sum the basic weight of the network communication mode and the urgency correction weight to obtain the dynamic weight of the network communication mode, and determine the hardware resource allocation ratio of each network communication mode according to the dynamic drive of each network communication mode.

[0173] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the above-mentioned log transmission method embodiment.

[0174] Please refer to Figure 5 , Figure 5 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. The embodiment of the present invention provides an electronic device 10, including a processor 11 and a memory 12; wherein the memory 12 is used to store computer programs; the processor 11 is used to execute the log transmission method provided by the aforementioned embodiment when executing the computer program.

[0175] For the specific process of the above-mentioned log transmission method, please refer to the corresponding content provided in the above-mentioned embodiment, which will not be repeated here.

[0176] Furthermore, the memory 12 as a carrier for resource storage may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0177] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16; wherein the power supply 13 is used to provide operating voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0178] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned log transmission method embodiments when running.

[0179] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0180] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned log transmission method embodiments are implemented.

[0181] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned log transmission method embodiments are implemented.

[0182] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] The above is a detailed introduction to a log transmission method, device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A log transmission method, characterized in that: include: Acquiring log data, determining the log data to be transmitted by each of the at least two network communication modes, and determining a transmission priority of each of the log data; Obtaining hardware resource usage of at least two types of hardware resources, determining an overall pressure value of the local device based on the hardware resource usage, and selecting a target transmission configuration that matches the overall pressure value from at least two preset transmission configurations corresponding to each of the network communication modes; Determining the hardware resource allocation ratio for each of the network communication modes based on the hardware resource usage of each of the hardware resource types, the types of hardware resources that each of the network communication modes relies on, and the transmission priority of the log data required to be transmitted by each of the network communication modes, and determining the hardware resource allocation amount for each of the network communication modes based on the hardware resource amount of each hardware resource type and the hardware resource allocation ratio; According to the target transmission configuration and hardware resource allocation of each network communication mode, the log data is transmitted in parallel using each network communication mode.

2. The log transmission method according to claim 1, characterized in that: Determining the overall pressure value of the machine according to the hardware resource usage includes: Obtaining preset initial weights for various hardware resource types, and obtaining at least two historical hardware resource usages corresponding to the various hardware resource types; Determining whether the historical hardware resource usage of the hardware resource type is greater than a preset threshold; If the historical hardware resource usage of the hardware resource type is greater than the preset threshold, the preset initial weight of the hardware resource type is increased to obtain the weighted weight of the hardware resource type; If the historical hardware resource usage of the hardware resource type is not greater than the preset threshold, the preset initial weight of the hardware resource type is used as the weighted weight of the hardware resource type; The overall pressure value is obtained by performing weighted processing on the usage of each hardware resource according to the weighted weight of each hardware resource type.

3. The log transmission method according to claim 2, characterized in that: The selecting, from the at least two preset transmission configurations corresponding to each of the network communication modes, a target transmission configuration that matches the overall pressure value includes: Determining a target overall pressure value interval in which the overall pressure value lies in at least two preset overall pressure value intervals; Among the preset transmission configurations corresponding to the network communication mode, a preset transmission configuration matching the target overall pressure value interval is determined as the target transmission configuration.

4. The log transmission method according to claim 3, characterized in that: The overall pressure value is periodically detected; The method further comprises: Obtaining an overall pressure target value corresponding to the target overall pressure value interval; Calculating the difference between the overall pressure target value and the overall pressure value to obtain a first control value at the current detection moment; Obtaining first control values ​​at multiple historical detection moments, summing the first control value at the current detection moment with the first control values ​​at each of the historical detection moments, and multiplying the sum by the detection cycle length to obtain a second control value; Determine the difference between the first control value at the current detection moment and the first control value at the most recent historical detection moment, and divide the difference result by the detection cycle length to obtain a third control value; weighting the first control value, the second control value, and the third control value to obtain a thread number adjustment value; The current number of threads of the local machine is adjusted according to the thread number adjustment value.

5. The log transmission method according to claim 1, wherein: Determining the transmission priority of each log data includes: Extracting a field to be detected from the log data, matching the field to be detected with a preset field of each transmission priority, and determining the transmission priority of the log data according to the matching result; And / or, determining the transmission priority of the log data according to the data source of the log data.

6. The log transmission method according to claim 1, wherein: Also includes: Determine whether the ring buffer is full; If the circular buffer area is not full, adding the log data to the circular buffer area; If the ring buffer is full, adding the log data to the memory; The adopting of the various network communication modes to transmit the log data in parallel includes: The log data is read from the ring buffer area or the memory, and the log data is transmitted in parallel using the various network communication modes.

7. The log transmission method according to any one of claims 1 to 6, characterized in that: The determining of the hardware resource allocation ratio of each network communication mode according to the hardware resource usage of each network communication mode, the type of hardware resources that each network communication mode relies on, and the transmission priority of the log data required to be transmitted by each network communication mode includes: Obtaining a preset initial weight for each of the network communication modes; For each of the hardware resource types, determining whether the hardware resource usage of the hardware resource type is greater than a preset threshold; If the hardware resource usage of the hardware resource type is not greater than the preset threshold, then for the network communication mode that depends on the hardware resource type, using the preset initial weight of the network communication mode as the basic weight of the network communication mode; If the hardware resource usage of the hardware resource type is greater than the preset threshold, a preset resource pressure coefficient is obtained, and for a network communication method that depends on the hardware resource type, a preset initial weight of the network communication method is lowered using the preset resource pressure coefficient to obtain a basic weight of the network communication method; the value of the preset resource pressure coefficient is between 0 and 1; Determining an urgency correction weight for each network communication mode according to the transmission priority of the log data transmitted by each network communication mode; the weight value of the urgency correction weight is proportional to the transmission priority; The basic weight of the network communication mode and the urgency correction weight are summed to obtain the dynamic weight of the network communication mode, and the hardware resource allocation ratio of each network communication mode is determined according to the dynamic driving of each network communication mode.

8. A log transmission device, characterized in that: include: a log processing module, configured to obtain log data, determine the log data to be transmitted by each of at least two network communication modes, and determine the transmission priority of each of the log data; a configuration adjustment module, configured to obtain hardware resource usage of at least two hardware resource types, determine an overall pressure value of the local device based on the hardware resource usage, and select a target transmission configuration that matches the overall pressure value from at least two preset transmission configurations corresponding to each of the network communication modes; a resource allocation module, configured to determine a hardware resource allocation ratio for each of the network communication modes based on the hardware resource usage of each of the network communication modes, the type of hardware resources that each of the network communication modes relies on, and the transmission priority of the log data required to be transmitted by each of the network communication modes, and to determine a hardware resource allocation amount for each of the network communication modes based on the hardware resource amounts of each type of hardware resources and the hardware resource allocation ratio; The log transmission module is used to adopt each of the network communication modes to transmit the log data in parallel according to the target transmission configuration and hardware resource allocation of each of the network communication modes.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the log transmission method according to any one of claims 1 to 7 when executing the computer program.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the log transmission method according to any one of claims 1 to 7 is implemented.

Images