Terminal log uploading method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of computer networks, and discloses a terminal log uploading method and device, computer equipment and a storage medium. The method comprises the following steps: when a log data stream is uploaded, repeatedly executing the following compression operations: monitoring a network condition, and determining a network condition adjustment parameter according to a network bandwidth and a network delay; determining the size of a sliding window according to the current remaining memory and the total memory; compressing the data block corresponding to the compression operation according to a compression ratio determined by the network condition adjustment parameter and the sliding window size to obtain a compressed data block, and integrating the compressed data block to obtain an integrated data block; and when it is determined that the to-be-uploaded data does not exist, determining the finally obtained integrated data block as a compressed data stream. According to the method, the log data stream can be subjected to block compression in the uploading process, the compression ratio is adjusted in real time according to the network condition and the memory condition, and the size of the sliding window is adjusted in real time according to the memory condition so as to adjust the size of the compressed data block.

Description

Technical Field

[0001] The present invention relates to the field of computer network technology, and in particular to a terminal log uploading method, device, computer equipment and storage medium. Background Art

[0002] With the popularization and functional expansion of smart terminal devices, the large amount of log data generated during their operation is of great significance for device performance analysis, troubleshooting, and user behavior research. However, there are many problems with the traditional log upload method:

[0003] Low upload efficiency: Traditional technologies often use simple fixed-size blocks or no blocks when processing log data, which results in large data packets when uploading large amounts of data. This not only prolongs the transmission time, but also easily leads to transmission failures when the network is poor.

[0004] High resource usage: Since data packets are large and not efficiently compressed, they occupy a large amount of network bandwidth when uploaded, which not only interferes with the normal use of terminal devices, but also has an adverse impact on other network applications.

[0005] Therefore, there is an urgent need for a terminal log uploading method that can improve transmission efficiency and reduce resource usage. Summary of the invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problems of long log upload time, low efficiency and high resource occupation in the related art.

[0007] In order to solve the above technical problems, the present invention provides a terminal log uploading method, comprising:

[0008] When uploading a log data stream, repeatedly perform the following compression operations until a compressed data stream corresponding to the log data stream is obtained:

[0009] Monitor the network status and obtain the network bandwidth and network delay targeted by this compression operation;

[0010] Determine the network condition adjustment parameter for this compression operation according to the network bandwidth and network delay for this compression operation;

[0011] Determine the sliding window size for this compression operation according to the current remaining memory and the total memory; the current remaining memory represents the current remaining memory of the terminal device; the total memory represents the total memory of the terminal device;

[0012] According to the network status of the compression operation, the parameters are adjusted and the sliding window size of the compression operation is determined to determine the compression rate of the compression operation;

[0013] Compressing the data block corresponding to the sliding window size targeted by the current compression operation according to the compression rate targeted by the current compression operation to obtain the compressed data block targeted by the current compression operation, and determining the compressed data block targeted by the current compression operation and the integrated data block targeted by the previous compression operation as the integrated data block targeted by the next compression operation;

[0014] When it is determined that there is no data block corresponding to the sliding window size targeted by the current compression operation, the integrated data block targeted by the previous compression operation is determined as the compressed data stream.

[0015] In an optional embodiment, the data block corresponding to the sliding window size targeted by the current compression operation is the log data that can be covered by the sliding window size targeted by the current compression operation, with the end point of the data block corresponding to the sliding window size targeted by the previous compression operation as the starting point; the data block corresponding to the sliding window size targeted by the first compression operation is the log data that can be covered by the sliding window size targeted by the first compression operation, with the starting point of the log data stream as the starting point.

[0016] In an optional implementation manner, determining the network condition adjustment parameter for the current compression operation according to the network bandwidth and network delay for the current compression operation includes:

[0017] The network condition adjustment parameter for this compression operation is determined according to the ratio of the network bandwidth and the network delay for this compression operation.

[0018] In an optional implementation, determining the sliding window size for the current compression operation according to the current remaining memory and the total memory includes:

[0019] The sliding window size for this compression operation is determined according to the ratio of the remaining memory to the total memory for this compression operation.

[0020] In an optional implementation, determining the sliding window size for the current compression operation according to the ratio of the remaining memory to the total memory for the current compression operation includes:

[0021] The sliding window size for this compression operation is determined according to the following size determination function:

[0022]

[0023] Wherein, W represents the sliding window size for this compression operation, P represents the sliding window adjustment parameter determined according to the ratio of the current remaining memory to the total memory, m represents the current remaining memory, and t represents the total memory.

[0024] In an optional implementation manner, the step of adjusting the parameter according to the network status of the compression operation and the sliding window size of the compression operation to determine the compression rate of the compression operation includes:

[0025] The compression ratio for this compression operation is determined according to the following compression ratio determination function:

[0026]

[0027] Among them, θ represents the compression rate of this compression operation, A(N,W) represents the compression rate determination function, N represents the network condition adjustment parameter of this compression operation, and W represents the sliding window size of this compression operation.

[0028] In an optional embodiment, the method further includes:

[0029] Predetermine the initial sliding window size;

[0030] Analyze the log data that is not currently uploaded to determine the data duplication;

[0031] The initial sliding window size is adjusted according to the data repetition degree to obtain the sliding window size targeted by this compression operation.

[0032] In a second aspect, the present invention provides a terminal log uploading device, comprising:

[0033] A first processing module, used for repeatedly performing a compression operation when uploading a log data stream until a compressed data stream corresponding to the log data stream is obtained;

[0034] The first processing module includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit and a fifth processing unit;

[0035] The first processing unit is used to monitor the network status and obtain the network bandwidth and network delay targeted by the current compression operation; determine the network status adjustment parameter targeted by the current compression operation according to the network bandwidth and network delay targeted by the current compression operation;

[0036] The second processing unit is used to determine the sliding window size for this compression operation according to the current remaining memory and the total memory; the current remaining memory represents the current remaining memory of the terminal device; the total memory represents the total memory of the terminal device;

[0037] The third processing unit is used to adjust the parameters according to the network status of the compression operation and the sliding window size of the compression operation to determine the compression rate of the compression operation;

[0038] The fourth processing unit is used to compress the data block corresponding to the sliding window size targeted by the current compression operation according to the compression rate targeted by the current compression operation to obtain the compressed data block targeted by the current compression operation, and determine the compressed data block targeted by the current compression operation and the integrated data block targeted by the previous compression operation as the integrated data block targeted by the next compression operation;

[0039] The fifth processing unit is configured to determine the integrated data block targeted by the last compression operation as the compressed data stream when it is determined that there is no data block corresponding to the sliding window size targeted by the current compression operation.

[0040] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the terminal log uploading method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a single computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the terminal log uploading method of the first aspect or any corresponding embodiment thereof.

[0042] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the terminal log uploading method of the first aspect or any corresponding embodiment thereof.

[0043] The technical solution provided by the present invention has the following technical effects:

[0044] Improving upload efficiency:

[0045] Dynamically adapt to network conditions: By monitoring the network bandwidth and network delay during each compression operation to determine the network condition adjustment parameters, the compression rate can be flexibly adjusted according to the real-time changes in the network environment. For example, when the network bandwidth is narrow and the delay is high (poor network conditions), the compression rate can be increased accordingly, so that the amount of data to be uploaded is greatly reduced, thereby shortening the transmission time and avoiding transmission failures or long-term freezes due to poor network conditions; when the network conditions are good, appropriately reducing the compression rate and reducing unnecessary compression calculation overhead will also help improve the overall upload efficiency and allow log data to be transmitted to the terminal device faster. Taking a practical scenario as an example, if the terminal device is in a wireless network environment with a weak signal, the network bandwidth may be only a few hundred KB per second. At this time, based on the low bandwidth and high delay detected, the compression rate is increased, and the originally large amount of data is compressed into smaller data packets for transmission, effectively utilizing limited network resources and speeding up upload speed.

[0046] Optimize data block processing: compress data blocks according to the size of the sliding window, and the sliding window size can be dynamically adjusted according to the current remaining memory and total memory of the terminal device, ensuring that the size of each compressed data block is compatible with the device memory resources. In this way, the memory can be fully utilized to reasonably select the data block range for efficient compression, avoiding situations where the data block is too large to be effectively compressed due to insufficient memory, or the data block is too small to increase unnecessary compression times, etc., which optimizes the data compression and processing process as a whole, and indirectly improves the upload efficiency.

[0047] Reasonable use of resources: The sliding window size is dynamically determined based on the current remaining memory and total memory of the terminal device. This means that when memory resources are sufficient, the sliding window size can be appropriately increased, and the amount of compressed log data can be increased accordingly, improving the compression effect, but it will not exceed the device memory capacity. When the memory is tight, the sliding window size is reduced accordingly to avoid affecting the operation of other functions of the device due to excessive memory occupation by an overly large window. For example, when the terminal device is running multiple large applications at the same time and there is less remaining memory, the sliding window size is reduced to ensure that the compression operation does not cause memory overflow, ensuring the overall stable operation of the device and achieving reasonable allocation and efficient use of memory resources.

[0048] Effective use of resources: The compression rate is determined based on the network conditions and the sliding window size, so that the intensity of the compression operation (compression rate) matches the current operating environment of the terminal device. Avoid using too high a compression rate when the network conditions allow, thereby reducing the excessive consumption of CPU resources due to complex compression algorithms, allowing the CPU to have more resources to handle other tasks on the terminal device, preventing the terminal device from freezing due to continuous high-load compression of log data, and improving the efficiency of CPU resource utilization.

[0049] Improve adaptability and reliability:

[0050] Adaptability to changing environments: The technical solution of the present invention dynamically adjusts relevant parameters according to the network status and the memory status of the terminal device during each compression operation, so that the upload of log data can be well adapted to different network environments (such as switching from a high-speed wired network to an unstable mobile network) and the resource status of the terminal device at different times (changes in memory and CPU load, etc.). Regardless of how the external conditions change, log data can be compressed and uploaded in the best possible way, which enhances the adaptability of the technical solution of the present invention in complex and changeable actual usage scenarios.

[0051] Ensure data integrity and accuracy: The next integrated data block is determined by merging the compressed data block obtained from each compression operation with the last integrated data block. When it is determined that there is no data block corresponding to the sliding window size corresponding to this compression operation (equivalent to the absence of log data to be compressed), the last integrated data block is determined as the final compressed data stream. Such a rigorous processing flow ensures that all data in the log data stream can be processed and compressed in an orderly and complete manner, avoiding data loss or confusion, improving the reliability of data upload, and facilitating subsequent accurate equipment performance analysis, troubleshooting, and other operations based on these log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a flowchart of a terminal log uploading method according to an embodiment of the present invention;

[0054] Figure 2 is a flow chart of another terminal log uploading method according to an embodiment of the present invention;

[0055] Figure 3 It is a structural schematic diagram of a terminal log uploading device according to an embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] Disadvantages of traditional technical solutions:

[0059] Inefficient upload:

[0060] On the one hand, the existing technology mostly uses simple fixed-size block or no block when processing log data, which causes the data packet size to be too large when uploading a large amount of data, which not only prolongs the transmission time but also easily causes transmission failure when the network is poor.

[0061] On the other hand, although some technologies are used to compress log data, the compression algorithms used are inefficient or not optimized for the characteristics of log data, resulting in a low compression ratio and limited effect on improving upload efficiency.

[0062] High resource usage: Since the data packets are large and not efficiently compressed, they take up a lot of network bandwidth when uploaded, which not only interferes with the normal use of the terminal device, but also has an adverse effect on other network applications. At the same time, during the upload process, device resources such as CPU, memory and battery are consumed in large quantities. Continuously uploading a large amount of log data may cause device performance to degrade, seriously affecting the user experience.

[0063] The embodiments of the present invention provide a terminal log uploading method, device, computer equipment and storage medium to solve the above problems, improve the upload speed, reduce resource occupation, adapt to changeable network conditions, and realize log data uploading efficiently and intelligently.

[0064] According to an embodiment of the present invention, an embodiment of a terminal log uploading method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] Figure 1 It is a flowchart of a terminal log uploading method according to an embodiment of the present invention.

[0066] like Figure 1 As shown, an embodiment of the present invention provides a terminal log uploading method, and the terminal log uploading method includes:

[0067] S101: When uploading a log data stream, repeatedly perform the following compression operations until a compressed data stream corresponding to the log data stream is obtained:

[0068] S1011: Monitor the network status, obtain the network bandwidth and network delay targeted by the current compression operation, and determine the network status adjustment parameters targeted by the current compression operation according to the network bandwidth and network delay targeted by the current compression operation.

[0069] In the present embodiment, the log data stream is the log data to be uploaded, and for the compression operation, it is the log data to be compressed. The technical solution of the present invention compresses and uploads the log data stream in blocks by repeatedly performing the compression operation, and obtains the complete compressed log data by integrating it into the memory of the terminal device, thereby realizing the uploading of the log data stream. When the compressed data stream corresponding to the log data stream is obtained, it indicates that the log data stream has been uploaded. The network bandwidth and network delay targeted by the first compression operation are the network bandwidth and network delay obtained by monitoring the network status when the log data stream starts to be uploaded.

[0070] In this embodiment, a network status monitoring module may be integrated to monitor the network status and collect network status information in real time, such as network bandwidth and network delay, to provide data support for uploading log data.

[0071] In this embodiment, the log data stream may be the following types of log data:

[0072] System log: The operating system of the terminal device generates a large amount of log data streams. For example, the event viewer of the Windows system records logs of various events such as system startup, shutdown, application installation, and device driver failure. These logs are continuously generated in the form of data streams, containing information such as the time, type, and source of the event. In the Linux system, various log files in the / var / log directory are common, such as syslog to record system-level information, and auth.log to record events related to user authentication.

[0073] Application logs: Various applications on terminal devices also generate log data streams. For example, web server software such as Apache and Nginx will record client access requests, response status codes, access times, and other information. These log data streams can help website administrators analyze website access and understand user behavior, such as which pages are most popular and which pages have more errors.

[0074] Network device logs: Terminal devices such as routers, switches, etc. also generate log data streams. These logs include information such as the establishment and disconnection of network connections, IP address allocation, and network traffic anomalies.

[0075] Security device logs: Security devices such as firewalls, intrusion detection systems (IDS) and intrusion prevention systems (IPS) generate log data streams. Firewall logs record the implementation of network access control policies, which IP addresses are allowed or denied access, etc. IDS and IPS logs record the detection and prevention of potential intrusion behaviors, such as whether malware propagation, port scanning and other attack behaviors are detected.

[0076] In this embodiment, determining the network condition adjustment parameter for this compression operation according to the network bandwidth and network delay for this compression operation in S1011 specifically includes:

[0077] The network condition adjustment parameter for this compression operation is determined according to the ratio of the network bandwidth and the network delay for this compression operation.

[0078] As an example, the network condition adjustment parameters for this compression operation are determined according to the following parameter determination function:

[0079] Among them, N represents the network condition adjustment parameter for this compression operation, k represents the network bandwidth for this compression operation, and d represents the network delay for this compression operation. According to the above formula, the smaller the N value is when the network condition is poor, the larger the N value is when the network condition is good.

[0080] S1012: Determine the sliding window size for this compression operation according to the current remaining memory and the total memory.

[0081] In this embodiment, the current remaining memory indicates the current remaining memory of the terminal device. The total memory indicates the total memory of the terminal device. The terminal device includes but is not limited to a smart phone, a tablet computer, an embedded device, etc.

[0082] In this embodiment, in order to ensure the system operation of the terminal device, a part of the memory needs to be reserved for system processes and tasks, and the part of the memory reserved for system processes and tasks is the reserved memory. Therefore, the current remaining memory in this embodiment is the part of the actual remaining memory of the terminal device excluding the reserved memory. As an example, the current remaining memory m = total memory t-used memory a-reserved memory b.

[0083] As an example, the sliding window size for this compression operation is determined according to the following size determination function: Wherein, W represents the sliding window size for this compression operation, P represents the sliding window adjustment parameter determined according to the ratio of the current remaining memory to the total memory, m represents the current remaining memory, and t represents the total memory.

[0084] S1013: Determine the compression rate for this compression operation according to the network status adjustment parameters for this compression operation and the sliding window size for this compression operation.

[0085] In this embodiment, the parameter N and the sliding window size W can be adjusted according to the network status, and the compression rate can be determined using the dynamic adjustment algorithm A. In this embodiment, the compression rate represents the compression level of the compression algorithm C, and the compression level and the compression rate are positively correlated. The smaller the compression level, the lower the compression rate, and the larger the compression level, the higher the compression rate.

[0086] In this embodiment,

[0087] As an example, the dynamic adjustment algorithm A determines the compression rate for this compression operation according to the following compression rate determination function:

[0088]

[0089] Among them, θ represents the compression rate of this compression operation, A(N,W) represents the compression rate determination function, N represents the network condition adjustment parameter of this compression operation, and W represents the sliding window size of this compression operation.

[0090] S1014: Compress the data block corresponding to the sliding window size targeted by this compression operation according to the compression rate targeted by this compression operation to obtain the compressed data block targeted by this compression operation, and determine the compressed data block targeted by this compression operation and the integrated data block targeted by the previous compression operation as the integrated data block targeted by the next compression operation.

[0091] In this embodiment, the data block corresponding to the sliding window size targeted by this compression operation is the log data that can be covered by the sliding window size targeted by this compression operation, starting from the end point of the data block corresponding to the sliding window size targeted by the last compression operation. The data block corresponding to the sliding window size targeted by the first compression operation is the log data that can be covered by the sliding window size targeted by the first compression operation, starting from the starting point of the log data stream.

[0092] In this embodiment, before executing S1014 to compress the data blocks corresponding to the sliding window size targeted by this compression operation according to the compression rate targeted by this compression operation, it is necessary to determine the data blocks corresponding to the sliding window size targeted by this compression operation. The data blocks corresponding to the sliding window size targeted by this compression operation are the log data that can be covered by the sliding window size.

[0093] In this embodiment, the compression rate of the compression algorithm C is adjusted according to the compression rate of this compression operation, and the compression algorithm C(θ) of this compression operation is obtained. The compression algorithm C(θ) of this compression operation is used to compress the data block d i (represents the data block corresponding to the sliding window size targeted by this compression operation) is compressed to obtain the compressed data block d′ i (indicates the compressed data block targeted by this compression operation), the specific d′i =C(θ)(d i ), i represents the index of the data block, corresponding to the i-th compression operation. i Added to the integrated data block. Compression algorithm C is the target compression algorithm used when the technical solution of the present invention performs compression.

[0094] The improvement of the present invention is not the improvement of a specific compression algorithm, but the determination of the compression rate of the compression algorithm. Therefore, a conventional compression algorithm in the art can be used as the compression algorithm C in the present invention. The compression algorithm C is preferably a real-time compression algorithm so that the compression can be completed while the log data is generated, and it will not become a performance bottleneck. As an example, the compression algorithm C can use a real-time compression algorithm such as the LZ77 algorithm and the LZ4 algorithm.

[0095] The technical solutions of S1011-S1013 mentioned above are the dynamic adjustment mechanism of the present invention. In S1014, the data blocks corresponding to the sliding window size targeted by the compression operation are compressed according to the compression rate targeted by the compression operation as a compression strategy. The present invention can dynamically adjust the compression strategy according to the dynamic adjustment mechanism.

[0096] S1015: When it is determined that there is no data block corresponding to the sliding window size targeted by the current compression operation, the integrated data block targeted by the previous compression operation is determined as the compressed data stream.

[0097] In this embodiment, the absence of a data block corresponding to the sliding window size targeted by this compression operation indicates that the log data stream is completely compressed and uploaded, and there is no log data to be compressed. When there is no log data to be compressed, the final integrated data block is determined as the compressed data stream.

[0098] In this embodiment, the data block corresponding to the sliding window size for the first compression operation is the entire log data stream D. For the above situation, only one compression operation is required. Specifically, D ′ =C(θ)(D),D ′ Indicates the compressed log data corresponding to the log data stream D.

[0099] In the technical solution of the present invention, compression is performed simultaneously during the process of uploading the log data stream, so that the log data uploaded to the terminal device is a compressed log data stream, that is, a compressed data stream. In the technical solution of the present invention, the log data uploading process and the data compression process are closely combined, and after the log data is generated, it can be understood that compression is performed simultaneously during the process of transmitting from the location where the log data is stored to the memory of the terminal device.

[0100] In the technical solution of the present invention, the entire log data stream can be divided into at least one data block by the sliding window size determined in real time, and a single compression operation compresses at least a part of the log data divided in the log data stream, thereby realizing block compression and reducing resource (network bandwidth, etc.) occupancy, and when determining the amount of compressed data corresponding to a single compression operation, the network condition and the memory condition of the terminal device are comprehensively considered, and the current network condition and the current memory condition are reasonably utilized while increasing the upload speed of the log data. When the network state is poor, the compression rate is increased to realize compression of a large amount of log data while increasing the upload speed. When the network state is good (the upload speed can be guaranteed), the compression rate is reduced to reduce the pressure of the compression algorithm while ensuring the upload speed.

[0101] As an example, in the technical solution of the present invention, when the current remaining memory is the same and the network conditions are different, when the network condition is poor (the network bandwidth is small and the network delay is large), the network condition adjustment parameter N is small, and the corresponding compression rate is high, so that the amount of data to be uploaded (the amount of data to be uploaded is the amount of data in the compressed data block after compression) is greatly reduced, thereby shortening the transmission time. When the network condition is good (the network bandwidth is large and the network delay is small), the network condition adjustment parameter N is large, and the corresponding compression rate is low. Although the amount of data to be uploaded (the amount of data to be uploaded is the amount of data in the compressed data block after compression) is large, due to the good network condition, a faster upload speed can still be guaranteed.

[0102] Dynamically adjusting the compression ratio can achieve the following effects from the perspective of resource utilization:

[0103] CPU resource saving: When the network is in good condition, lowering the compression ratio can reduce the CPU usage of compression operations. Compression algorithms with high compression ratios are usually more complex and require a lot of calculations to find repeated patterns in the data, perform complex encoding and replacement operations, etc. For example, some advanced compression algorithms may use multi-layer nested dictionary structures to find the optimal compression method, which consumes a lot of CPU time. A lower compression ratio means a simpler compression algorithm with greatly reduced calculations, thus freeing up CPU resources for other tasks, such as running other applications, performing system updates, or background data processing.

[0104] Memory resource release: A high compression rate compression process may require more memory to store intermediate data and complex compression structures. For example, some compression algorithms will build a large hash table to quickly locate repeated patterns at high compression rates, which will take up a lot of memory space. When the compression rate is reduced, these additional memory usage can be alleviated, allowing memory to be used for other more needed places, such as caching more application data or system files, thereby improving the overall performance of the device.

[0105] Dynamically adjusting the compression ratio can achieve the following effects from the perspective of data processing speed:

[0106] Faster compression: Lower compression rates generally mean faster compression. Compression with high compression rates may require multiple scans and complex transformations of the data, but with lower compression rates, the compression algorithm can process data blocks faster. For example, for some compression algorithms based on pattern matching, only simple local pattern matching is required at low compression rates, without the need for deep searches in the entire data set. This significantly shortens the compression time for each data block, thereby improving the overall data processing speed.

[0107] Improved decompression speed: On the receiving end, a lower compression rate also means faster decompression speed. When the uploaded log data needs to be decompressed and analyzed, the data with low compression rate can be restored to its state faster. Highly compressed data may require complex decoding and reorganization processes when decompressed, while low-compression data is relatively simple to decompress and can provide usable log data faster, which is beneficial for subsequent data analysis and troubleshooting operations.

[0108] Dynamically adjusting the compression ratio can also achieve the following effects:

[0109] Avoid resource competition: If you maintain a high compression rate when the network is in good condition, the device resources may be overly concentrated on the compression task, which may lead to resource competition. For example, the device's response speed may slow down, affecting other operations that the user is doing, such as opening applications and browsing the web. Reducing the compression rate can avoid this situation, allowing the device to maintain good response performance while uploading log data, improving the user experience.

[0110] Reduce the risk of system failure: Continuous high compression rate operation may increase the risk of system failure. For example, excessive use of CPU and memory resources may cause device overheating, application crashes, or system instability. By reducing the compression rate when the network status is good, this risk can be reduced and the stable operation of the device can be ensured.

[0111] Figure 2 The figure is a flowchart of another terminal log uploading method according to an embodiment of the present invention.

[0112] like Figure 2 As shown, another terminal log uploading method provided by an embodiment of the present invention includes:

[0113] S201: When uploading a log data stream, repeatedly perform the following compression operations until a compressed data stream corresponding to the log data stream is obtained:

[0114] S2011: Monitor the network status, obtain the network bandwidth and network delay for this compression operation, and determine the network status adjustment parameters for this compression operation according to the network bandwidth and network delay for this compression operation. For details, refer to the relevant description of S1011, which will not be repeated here.

[0115] S2012: predetermine an initial sliding window size, analyze the currently unuploaded log data to determine data duplication, and adjust the initial sliding window size according to the data duplication to obtain a sliding window size for this compression operation.

[0116] In this embodiment, the currently unuploaded log data represents the currently unuploaded portion of the log data stream, and the currently uploaded log data is the log data that has been compressed before this compression operation. For the first compression operation, the currently unuploaded log data is the log data stream. For a single compression operation, the data duplication of this compression operation can be determined by analyzing the currently unuploaded log data.

[0117] In this embodiment, a retrieval sliding window may be set, and the data duplication may be determined by analyzing the log data that is not currently uploaded through the retrieval sliding window.

[0118] As an example, the specific implementation method may be:

[0119] Data partitioning and window sliding: First, treat the currently unuploaded log data as a continuous data stream. According to the set retrieval sliding window size, divide the data blocks from the starting position of the unuploaded log data. For example, suppose the unuploaded log data is a system operation record in text form, and the retrieval sliding window size is set to 10 characters. The retrieval sliding window will start from the beginning of the log data stream, cover the first 10 characters as the first data block, and then slide forward one character each time (or according to the specific sliding step size), so that a series of partially overlapping data blocks will be generated in sequence. If the log data stream is "abcdefghijklmnopqrstuvwxyz", the first data block is "abcdefghijk", the second is "bcdefghijk", and so on.

[0120] Repeatability statistics:

[0121] Simple counting method: During the sliding process, the number of times each data block appears is counted. When the retrieval sliding window generates a new data block, it is compared with the data block that has appeared before. If the same data block is found, the count of the data block is increased by 1. For example, during the processing, it was found that the data block "abcdefghij" (repeating pattern) appeared 3 times, "bcdefghijk" appeared 2 times, and so on. Finally, the repetition of the data can be roughly determined based on the frequency of occurrence of each data block. Generally speaking, the greater the proportion of data blocks with higher frequency of occurrence, the higher the repetition of the data. The size of the retrieval sliding window can be a fixed value or it can be adjustable. For example, when the size of the retrieval sliding window is L value, no repeated data blocks are found. It may be that the L value is too large or too small. At this time, the size of the retrieval sliding window can be adjusted to re-count.

[0122] Similarity comparison method: In addition to counting exactly the same data blocks, the similarity between data blocks can also be considered. Some text similarity algorithms (such as the edit distance algorithm) can be used to measure the similarity between data blocks. For example, for the two data blocks "abcdefghij" and "abcdefghik", they only have one character different, and their similarity can be calculated through the edit distance algorithm. Then, a similarity threshold is set. When the similarity between the two data blocks exceeds this threshold, they are considered to be similar data blocks and are also included in the statistical range of repetition. This method can capture similar patterns in the data more flexibly and is more sensitive to repeated data with some minor changes.

[0123] Hash value-based method: Calculate a hash value for each data block. The hash function can convert a data block into a fixed-length hash code. When a new data block is generated, calculate its hash value and compare it with the hash value of the previous data block. If the hash values ​​are the same, it is likely that the data block contents are the same (although there is a small probability of hash collision). By counting the number of data blocks with the same hash value, the duplication of the data can be quickly determined. This method is more efficient when processing large amounts of data because comparing hash values ​​is usually faster than directly comparing the data block contents.

[0124] Comprehensive evaluation to determine duplication: Based on the above statistical results, the data duplication can be comprehensively evaluated. A simple way is to calculate a duplication index, for example, the sum of the number of occurrences of all duplicate data blocks (or similar data blocks) is divided by the total number of data blocks to obtain a duplication value between 0 and 1. The closer the value is to 1, the higher the duplication of the data. The closer the value is to 0, the lower the duplication. It is also possible to determine a more reasonable duplication evaluation standard based on actual application scenarios and data characteristics, combined with different statistical methods. For example, in log data with higher security requirements, more attention is paid to the duplication of exactly the same data blocks. In analysis scenarios with a certain tolerance for data changes, more consideration can be given to similar data blocks to determine data duplication.

[0125] As an example, a specific implementation method of adjusting the initial sliding window size according to the data repetition degree to obtain the sliding window size for this compression operation may be:

[0126] Limit the data repetition to the range of [0, 1], where 0 means completely random and 1 means completely repeated. You can use a function to dynamically adjust the initial sliding window size:

[0127] The initial sliding window size can be dynamically adjusted according to the specific calculation function: W=W0+(R×ΔW), where W represents the sliding window size for this compression operation, W0 represents the initial sliding window size, R represents the data duplication, and ΔW represents the maximum adjustable window size increment.

[0128] Or dynamically adjust the initial sliding window size according to the piecewise function:

[0129] Wherein, s represents a first preset value, and j represents a second preset value.

[0130] As an example, the maximum adjustable window size increment may be determined in the following manner:

[0131] Among them, n represents the current remaining memory, and n represents the space actually occupied by each byte of data in the memory. For example, assuming that each byte of data occupies 1 unit of space in the memory, then n=1.

[0132] S2013: According to the network status adjustment parameters and the sliding window size of the compression operation, the compression rate of the compression operation is determined. For details, please refer to the relevant description of S1013, which will not be repeated here.

[0133] S2014: Compress the data block corresponding to the sliding window size targeted by the current compression operation according to the compression rate targeted by the current compression operation to obtain the compressed data block targeted by the current compression operation, and determine the compressed data block targeted by the current compression operation and the integrated data block targeted by the previous compression operation as the integrated data block targeted by the next compression operation. For details, refer to the relevant description of S1014, which will not be repeated here.

[0134] S2015: When it is determined that there is no data block corresponding to the sliding window size targeted by the current compression operation, the integrated data block targeted by the previous compression operation is determined as the compressed data stream. For details, refer to the relevant description of S1015, which will not be repeated here.

[0135] In this embodiment, there is a situation in which the data block corresponding to the sliding window size targeted by the compression operation in S2014 is the entire log data stream. In this case, the compression operation is performed only once, and the corresponding technical solution is:

[0136] When uploading a log data stream, perform the following compression operations to obtain the compressed data stream corresponding to the log data stream:

[0137] Monitor the network status, obtain the current network bandwidth and the current network delay, and determine the current network status adjustment parameters according to the current network bandwidth and the current network delay.

[0138] The initial sliding window size is determined in advance, the log data stream is analyzed to determine the data repetition, and the initial sliding window size is adjusted according to the data repetition to obtain the adjusted sliding window size. Specifically, the initial sliding window size is determined in advance, a search sliding window is set, and the currently unuploaded log data is analyzed through the search sliding window to determine the data repetition. The data repetition is limited to the range of [0, 1], and a function is used to adjust the initial sliding window size to obtain the adjusted sliding window size.

[0139] The compression ratio is determined by adjusting the parameters and the adjusted sliding window size according to the current network conditions.

[0140] The log data stream is compressed according to the compression ratio to obtain a compressed data stream, thereby realizing the uploading of the log data stream.

[0141] The technical solution of the present invention can optimize real-time and dynamic compression: introduce an efficient real-time compression algorithm to compress log data while it is being generated, significantly reducing the amount of data transmission. Dynamically adjust the compression strategy, automatically adjust the compression rate according to the content and characteristics of the log data, balance the compression ratio and computing resource consumption, and achieve the best compression effect.

[0142] Real-time compression algorithms usually involve efficient online compression techniques, such as the LZ series of algorithms, which compress log data while it is generated to reduce storage and transmission costs.

[0143] Compression algorithm overview:

[0144] Input: Log data stream D (a series of bytes or data blocks).

[0145] Compression function: Use a real-time compression algorithm C (such as LZ77 algorithm or LZ4 algorithm, etc.) that supports setting a sliding window size to find repeated patterns in the input data stream.

[0146] Output: Compressed log data stream D′.

[0147] Dynamic compression optimization involves dynamically adjusting the compression strategy based on the characteristics of log data and network conditions.

[0148] Algorithm Overview:

[0149] Input: log data stream D, network status adjustment parameter N, and sliding window adjustment parameter P.

[0150] Compression strategy adjustment function: Use the dynamic adjustment algorithm A to obtain the compression rate θ of the compression algorithm C based on N and W.

[0151] Output: optimized compressed data stream D′.

[0152] Dynamic adjustment strategy:

[0153] Compression rate adjustment: When the network condition is poor, increase the compression rate to reduce the data volume. When the network condition is good, reduce the compression rate to reduce the computational overhead of compression and decompression.

[0154] Window size adjustment: Dynamically adjust the sliding window size in the compression algorithm based on the repetition and pattern of log data, or the memory status of the terminal device to optimize the compression effect.

[0155] It should be noted that dynamic compression optimization is a complex process that requires comprehensive consideration of multiple factors. The specific parameters of dynamic adjustment algorithm A and compression algorithm C may vary depending on the implementation. There is often a trade-off between real-time performance and compression effect, which needs to be compromised according to the actual application scenario.

[0156] The technical solution of the present invention can significantly improve the upload efficiency: reduce the size of the data packet through intelligent compression processing, optimize the upload process in combination with dynamic scheduling strategy, and significantly shorten the upload time. Reduce resource consumption: reduce unnecessary network bandwidth occupation and terminal resource consumption (such as CPU, memory, power, etc.), and improve the endurance of the device. Strong adaptability: can dynamically adjust the upload strategy according to changes in the network environment and device status, ensuring efficient upload under various conditions.

[0157] It should be noted that the contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

[0158] In the technical solution of the present invention, all the collection work involving customer identity information, credit data, etc., the prompt information of the loan application generated according to the prompt information template, the loan approval result, the violation prompt information, the repayment prompt information, the account status prompt information, etc., which are sent according to the preset method, are carried out in strict accordance with relevant laws, regulations and regulatory requirements. For example, before data collection, ensure that the customer's explicit authorization has been obtained, fully inform the customer of the purpose, scope, use method and data security measures of the data collection and other key information, and obtain data through legal and compliant channels and processes. At the same time, according to the relevant provisions and industry norms on personal information protection, establish a sound data collection management system and security protection system, encrypt and store the collected data, strictly control access and conduct regular security audits, so as to effectively protect the legality, security and integrity of customer data, so that the entire data collection process is completely within the framework of legality and compliance.

[0159] In this embodiment, a terminal log upload device is also provided, and a single device is used to implement the above-mentioned embodiment and optional implementation methods, which have been described and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0160] Figure 3 It is a structural diagram of a terminal log uploading device according to an embodiment of the present invention.

[0161] The present invention provides a terminal log uploading device, such as Figure 3 As shown, the terminal log uploading device includes:

[0162] The first processing module 11 is used to repeatedly perform a compression operation when uploading a log data stream until a compressed data stream corresponding to the log data stream is obtained.

[0163] The first processing module 11 includes a first processing unit 111 , a second processing unit 112 , a third processing unit 113 , a fourth processing unit 114 and a fifth processing unit 115 .

[0164] The first processing unit 111 is configured to monitor the network status, obtain the network bandwidth and network delay targeted by the current compression operation, and determine the network status adjustment parameter targeted by the current compression operation according to the network bandwidth and network delay targeted by the current compression operation.

[0165] The second processing unit 112 is used to determine the sliding window size for this compression operation according to the current remaining memory and the total memory. The current remaining memory refers to the current remaining memory of the terminal device. The total memory refers to the total memory of the terminal device.

[0166] The third processing unit 113 is used to adjust the parameters according to the network status of the current compression operation and the sliding window size of the current compression operation to determine the compression rate of the current compression operation.

[0167] The fourth processing unit 114 is used to compress the data block corresponding to the sliding window size targeted by the current compression operation according to the compression rate targeted by the current compression operation, to obtain the compressed data block targeted by the current compression operation, and to determine the compressed data block targeted by the current compression operation and the integrated data block targeted by the previous compression operation as the integrated data block targeted by the next compression operation.

[0168] The fifth processing unit 115 is configured to determine the integrated data block targeted by the last compression operation as the compressed data stream when it is determined that there is no data block corresponding to the sliding window size targeted by the current compression operation.

[0169] In an optional implementation, the data block corresponding to the sliding window size targeted by this compression operation is the log data that can be covered by the sliding window size targeted by this compression operation, starting from the end point of the data block corresponding to the sliding window size targeted by the last compression operation. The data block corresponding to the sliding window size targeted by the first compression operation is the log data that can be covered by the sliding window size targeted by the first compression operation, starting from the starting point of the log data stream.

[0170] In an optional implementation manner, the first processing unit 111 is specifically configured to determine a network condition adjustment parameter for the current compression operation according to a ratio of a network bandwidth and a network delay for the current compression operation.

[0171] In an optional implementation, the second processing unit 112 is specifically configured to determine the sliding window size for the current compression operation according to the ratio of the remaining memory to the total memory for the current compression operation.

[0172] In an optional implementation, the second processing unit 112 is specifically configured to determine the sliding window size for this compression operation according to the following size determination function:

[0173]

[0174] Wherein, W represents the sliding window size for this compression operation, P represents the sliding window adjustment parameter determined according to the ratio of the current remaining memory to the total memory, m represents the current remaining memory, and t represents the total memory.

[0175] In an optional implementation manner, the third processing subunit 113 is specifically configured to determine the compression rate for this compression operation according to the following compression rate determination function:

[0176]

[0177] Among them, θ represents the compression rate of this compression operation, A(N,W) represents the compression rate determination function, N represents the network condition adjustment parameter of this compression operation, and W represents the sliding window size of this compression operation.

[0178] In an optional implementation, the terminal log uploading device further includes: a size determination module, configured to predetermine an initial sliding window size.

[0179] Analyze the log data that is not currently uploaded to determine the data duplication.

[0180] The initial sliding window size is adjusted according to the data duplication to obtain the sliding window size targeted by this compression operation.

[0181] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0182] The terminal log uploading device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0183] The embodiment of the present invention also provides a computer device having the above Figure 3 The terminal log upload device is shown.

[0184] See also Figure 4 , Figure 4 Schematic diagram of the hardware structure of the computer device according to the embodiment of the present invention. Figure 4As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In an optional embodiment, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor device). Figure 4 A processor 10 is taken as an example.

[0185] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0186] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0187] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating device, an application required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In an optional embodiment, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0188] The memory 20 may include a volatile memory, such as a random access memory. The memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive. The memory 20 may also include a combination of the above types of memory.

[0189] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0190] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be downloaded through a network and stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0191] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0192] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A terminal log uploading method, characterized in that: include: When uploading a log data stream, repeatedly perform the following compression operations until a compressed data stream corresponding to the log data stream is obtained: Monitor the network status and obtain the network bandwidth and network delay targeted by this compression operation; Determine the network condition adjustment parameter for this compression operation according to the network bandwidth and network delay for this compression operation; Determine the sliding window size for this compression operation based on the current remaining memory and total memory; The current remaining memory represents the current remaining memory of the terminal device; the total memory represents the total memory of the terminal device; According to the network status of the compression operation, the parameters are adjusted and the sliding window size of the compression operation is determined to determine the compression rate of the compression operation; Compress the data block corresponding to the sliding window size targeted by the current compression operation according to the compression rate targeted by the current compression operation to obtain the compressed data block targeted by the current compression operation, and determine the compressed data block targeted by the current compression operation and the integrated data block targeted by the previous compression operation as the integrated data block targeted by the next compression operation; When it is determined that there is no data block corresponding to the sliding window size targeted by the current compression operation, the integrated data block targeted by the previous compression operation is determined as the compressed data stream.

2. The method according to claim 1, characterized in that The data block corresponding to the sliding window size targeted by the current compression operation is the log data that can be covered by the sliding window size targeted by the current compression operation, with the end point of the data block corresponding to the sliding window size targeted by the previous compression operation as the starting point; the data block corresponding to the sliding window size targeted by the first compression operation is the log data that can be covered by the sliding window size targeted by the first compression operation, with the starting point of the log data stream as the starting point.

3. The method according to claim 1, characterized in that Determining the network condition adjustment parameter for the compression operation according to the network bandwidth and network delay for the compression operation includes: The network condition adjustment parameter for this compression operation is determined according to the ratio of the network bandwidth and the network delay for this compression operation.

4. The method according to claim 1, characterized in that: The determining of the sliding window size for the compression operation according to the current remaining memory and the total memory includes: The sliding window size for this compression operation is determined according to the ratio of the remaining memory to the total memory for this compression operation.

5. The method according to claim 4, characterized in that The determining of the sliding window size for the compression operation according to the ratio of the remaining memory to the total memory for the compression operation includes: The sliding window size for this compression operation is determined according to the following size determination function: Wherein, W represents the sliding window size for this compression operation, P represents the sliding window adjustment parameter determined according to the ratio of the current remaining memory to the total memory, m represents the current remaining memory, and t represents the total memory.

6. The method according to claim 1, characterized in that The step of adjusting the parameters according to the network status of the compression operation and the sliding window size of the compression operation to determine the compression rate of the compression operation includes: The compression ratio for this compression operation is determined according to the following compression ratio determination function: Among them, θ represents the compression rate of this compression operation, A(N,W) represents the compression rate determination function, N represents the network condition adjustment parameter of this compression operation, and W represents the sliding window size of this compression operation.

7. The method according to claim 1, characterized in that The method further comprises: Predetermine the initial sliding window size; Analyze the log data that is not currently uploaded to determine the data duplication; The initial sliding window size is adjusted according to the data repetition degree to obtain the sliding window size targeted by this compression operation.

8. A terminal log uploading device, characterized in that: include: A first processing module, used for repeatedly performing a compression operation when uploading a log data stream until a compressed data stream corresponding to the log data stream is obtained; The first processing module includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit and a fifth processing unit; The first processing unit is used to monitor the network status and obtain the network bandwidth and network delay targeted by the current compression operation; and determine the network status adjustment parameter targeted by the current compression operation according to the network bandwidth and network delay targeted by the current compression operation; The second processing unit is used to determine the sliding window size for this compression operation according to the current remaining memory and the total memory; The current remaining memory represents the current remaining memory of the terminal device; the total memory represents the total memory of the terminal device; The third processing unit is used to adjust the parameters according to the network status of the compression operation and the sliding window size of the compression operation to determine the compression rate of the compression operation; The fourth processing unit is used to compress the data block corresponding to the sliding window size targeted by the current compression operation according to the compression rate targeted by the current compression operation to obtain the compressed data block targeted by the current compression operation, and determine the compressed data block targeted by the current compression operation and the integrated data block targeted by the previous compression operation as the integrated data block targeted by the next compression operation; The fifth processing unit is configured to determine the integrated data block targeted by the last compression operation as the compressed data stream when it is determined that there is no data block corresponding to the sliding window size targeted by the current compression operation.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the terminal log uploading method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the terminal log uploading method according to any one of claims 1 to 7.

Images