Log monitoring and compression method and device, computer device and storage medium

Abstract

This application relates to a log monitoring and compression method, apparatus, device, and storage medium, belonging to the field of data processing technology. The method includes: monitoring and acquiring a target log file and its first attribute information according to a preset time period; determining compression parameters for the target log file based on the first attribute information; compressing the target log file based on the compression parameters to obtain a target compressed file; and synchronously updating stored data in a memory based on the target compressed file. This application can achieve periodic monitoring and compression of log files corresponding to any component, and simultaneously perform periodic cleanup of the memory based on the compressed file, thereby improving memory utilization. While avoiding resource waste, it facilitates developers or maintenance personnel to quickly analyze and solve problems, thus reducing the time and economic costs of vehicle development.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a log monitoring and compression method, apparatus, device, and storage medium. Background Technology

[0002] In the whole vehicle components, it is necessary to record information on daily behavior and program abnormalities. There are two types of logs: system logs and maintenance logs. System logs record the process scheduling information generated during the operation of each system program, while maintenance logs record some status information of the component during operation using a standard JSON text format. Both types of logs can record some key information during the program operation or debugging phase.

[0003] When certain components malfunction, developers need to analyze and locate the problem to find a solution. The aforementioned log system is one way to quickly locate the problem. Since the disk resources of each component are limited, if the logs are written continuously, the number of log files will increase and the disk space of the components will be insufficient. When the external control module needs to subscribe to the logs periodically, if the subscribed logs are source files, it will have a significant impact on the transmission channel pressure. Existing technology usually uses the method of overwriting the earliest stored logs with the later collected log files. This solution will lead to the inability to accurately analyze the vehicle status, resulting in a waste of storage space resources and increasing the time and economic costs of vehicle development.

[0004] Therefore, there is an urgent need to propose a log monitoring and compression method, device, computer equipment, and storage medium that can avoid resource waste and reduce the time and economic costs of vehicle development. Summary of the Invention

[0005] Therefore, it is necessary to provide a log monitoring and compression method, device, computer equipment, and storage medium that can avoid resource waste and reduce the time and economic costs of vehicle development, in order to address the above-mentioned technical problems.

[0006] On the one hand, a log monitoring and compression method is provided, the method comprising:

[0007] According to a preset time period, monitor and obtain the target log file and the first attribute information of the target log file;

[0008] Based on the first attribute information of the target log file, the compression parameters of the target log file are determined;

[0009] Based on the compression parameters, the target log file is compressed to obtain a target compressed file;

[0010] Based on the target compressed file, the stored data in the memory is updated synchronously.

[0011] Optionally, before monitoring and obtaining the target log file and its first attribute information according to a preset time period, the method further includes:

[0012] Based on the data upload interface and the monitoring directory, the relevant parameters of the target compressed file are determined. The relevant parameters include at least one of the following: preset time period, source file format, compressed file format, compressed file name, and interface identifier.

[0013] Based on the second attribute information of the storage, a first limit size and a second limit size of the compressed file are determined. The second attribute information includes the storage capacity. The first limit size is the maximum size of the compressed file, and the second limit size is the maximum storage size occupied by the compressed file.

[0014] Based on the relevant parameters, the first limit size, and the second limit size, an external data object compressed file is generated for monitoring and compressing the target log file.

[0015] Optionally, the step of monitoring and obtaining the target log file and its first attribute information according to a preset time period includes:

[0016] In response to the detection of the start of a preset time period, the log files uploaded by the target upload interface within the preset time period are continuously monitored based on a real-time monitoring program.

[0017] In response to the detection that a single preset time period has ended, all log files within the preset time period are retrieved;

[0018] Based on the compressed file of the external data object, the target log file is selected from all log files, and the first attribute information of the target log file is obtained. The first attribute information includes at least one of the following: file format, size, interface identifier, scheduling flag, time period start timestamp, and time period end timestamp.

[0019] Optionally, determining the compression parameters of the target log file based on the first attribute information of the target log file includes:

[0020] Obtain the scheduling flag of the target log file to determine whether the real-time monitoring program is being started for the first time;

[0021] In response to the detection that the real-time monitoring program is being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file, and the flag bit corresponding to the scheduling flag is cleared.

[0022] In response to the detection that the real-time monitoring program is not being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file;

[0023] Based on the preset mapping table and pattern allocation strategy, the target log files in the first list are classified and saved to the second list;

[0024] Based on the first attribute information of the target log file in the second list, the corresponding external data object compressed file is matched to determine the compression parameters of the target log file.

[0025] Optionally, the preset mapping table includes at least one of the following: mapping name and priority code. The step of classifying and saving the target log files in the first list to the second list based on the preset mapping table and the mode allocation strategy includes:

[0026] Based on the interface identifier to which the target log file belongs, determine the mapping name corresponding to the target log file;

[0027] Based on the mapping name, determine the priority code of the target log file;

[0028] Based on the priority code, the mode to which the target log file belongs is determined, and the mode includes at least one of the following: basic mode, normal priority mode, and category mode;

[0029] Based on the pattern to which the target log files belong, multiple target log files are classified, and the target log files are saved to a second list according to the classification results.

[0030] Optionally, before compressing the target log file to obtain the target compressed file, the method further includes:

[0031] Based on the first size limit, determine whether the target log file meets the preset standard;

[0032] In response to the detection that the target log file meets a preset standard, the target log file is compressed to obtain a target compressed file.

[0033] Optionally, the step of synchronously updating the stored data in the memory based on the target compressed file includes:

[0034] Obtain the required space usage of the target compressed file;

[0035] Based on the required space occupancy, the memory is cleaned up, including:

[0036] In response to detecting that the target log file belongs to the basic mode, the storage is cleaned up according to the creation time order of the log files;

[0037] In response to detecting that the target log file belongs to a normal priority mode or a category mode, the memory is cleaned up according to the priority code of the log file;

[0038] In response to the detection that the required space occupancy exceeds a preset memory size for a consecutive preset number of time periods, the amount of space to be cleared is calculated using a preset algorithm, which includes:

[0039]

[0040] Where X represents the size of the space to be cleared, Y represents the first limit size, and N represents the cumulative value corresponding to the preset number of consecutive times;

[0041] Based on the value of the space to be cleaned, the memory is cleaned;

[0042] The target compressed file is saved to the cleaned storage space to synchronously update the stored data in the storage space.

[0043] On the other hand, a log monitoring and compression device is provided, the device comprising:

[0044] The monitoring and acquisition module is used to monitor and acquire the target log file and the first attribute information of the target log file according to a preset time period.

[0045] The compression parameter determination module is used to determine the compression parameters of the target log file based on the first attribute information of the target log file;

[0046] A compression module is used to compress the target log file based on the compression parameters to obtain a target compressed file;

[0047] The storage data update module is used to synchronously update the stored data in the memory based on the target compressed file.

[0048] In another aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0049] According to a preset time period, monitor and obtain the target log file and the first attribute information of the target log file;

[0050] Based on the first attribute information of the target log file, the compression parameters of the target log file are determined;

[0051] Based on the compression parameters, the target log file is compressed to obtain a target compressed file;

[0052] Based on the target compressed file, the stored data in the memory is updated synchronously.

[0053] In another aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0054] According to a preset time period, monitor and obtain the target log file and the first attribute information of the target log file;

[0055] Based on the first attribute information of the target log file, the compression parameters of the target log file are determined;

[0056] Based on the compression parameters, the target log file is compressed to obtain a target compressed file;

[0057] Based on the target compressed file, the stored data in the memory is updated synchronously.

[0058] The aforementioned log monitoring and compression method, apparatus, device, and storage medium, wherein the method includes: monitoring and acquiring a target log file and its first attribute information according to a preset time period; determining compression parameters for the target log file based on the first attribute information; compressing the target log file based on the compression parameters to obtain a target compressed file; and synchronously updating stored data in the memory based on the target compressed file. This application can realize periodic monitoring and compression of log files corresponding to any component, and simultaneously perform periodic cleaning of the memory based on the compressed file, thereby improving memory utilization. While avoiding resource waste, it facilitates developers or maintenance personnel to quickly analyze and solve problems, thereby reducing the time and economic costs of vehicle development. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating a log monitoring and compression method in one embodiment;

[0060] Figure 2 This is a schematic diagram illustrating an application scenario of the log monitoring and compression method in one embodiment;

[0061] Figure 3 This is a structural block diagram of the log monitoring and compression device in one embodiment;

[0062] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] It should be understood that, in the description of this application, unless the context explicitly requires it, words such as "including" or "comprising" throughout the specification should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0065] It should also be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0066] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0067] Example 1: In one example, as Figure 1 As shown, a log monitoring and compression method is provided, including the following steps:

[0068] S1: Monitor and obtain the target log file and its first attribute information according to a preset time period.

[0069] It should be noted that the preset time period refers to the time period for monitoring and compressing the target log file, which can be 1 hour, 1 day, etc. Users can set the specific value of this time period according to actual needs. The target log file can include system logs and maintenance logs of vehicle components. When certain components malfunction, developers can analyze and locate the problem through the corresponding log file to find a solution. The first attribute information of the target log file can include file format, size, interface identifier, scheduling flag, time period start timestamp, and time period end timestamp. The file format can be common text formats such as log and txt. The size refers to the required space. The interface identifier can include the component identifier (used to identify the source of the log file) and the external module interface (used to synchronize to a remote end or provide feedback to external developers or maintenance personnel). The scheduling flag is used to determine whether the real-time monitoring program is started for the first time (the flag is not set by default when the real-time monitoring program is started for the first time).

[0070] In some specific embodiments, before monitoring and obtaining the target log file and the first attribute information of the target log file according to a preset time period, the method further includes:

[0071] Based on the data upload interface and the monitoring directory, the relevant parameters of the target compressed file are determined. The relevant parameters include at least one of the following: preset time period, source file format, compressed file format, compressed file name, and interface identifier. The monitoring directory refers to the directory of log files to be monitored. The data upload interface is used to determine the interface identifier. The source file format is the file format mentioned above. The compressed file format refers to the compressed file format, such as zip, tgz, lz4, etc.

[0072] Based on the second attribute information of the storage, the first limit size and the second limit size of the compressed file are determined. The second attribute information includes the storage capacity. The first limit size refers to the maximum size limit of the compressed file (the maximum storage space occupied by a single compressed file, used to control the size of each compressed file). The second limit size refers to the maximum storage space occupied by the compressed file (the maximum storage space occupied by all compressed files, used for file cyclic overwriting). The first limit size and the second limit size can be set according to the storage capacity and actual needs.

[0073] Based on the relevant parameters, the first limit size, and the second limit size, an external data object compressed file is generated. This external data object compressed file is used as a constraint condition for monitoring and compressing the target log file.

[0074] In some specific implementations, the step of monitoring and acquiring the target log file and the first attribute information of the target log file according to a preset time period includes:

[0075] In response to the detection of the start of a preset time period, the system continuously monitors the log files uploaded by the target upload interface within the preset time period based on a real-time monitoring program (such as a log monitoring system).

[0076] In response to the detection that a single preset time period has ended, all log files within the preset time period are retrieved;

[0077] Based on the compressed file of the external data object, the target log file is selected from all log files, that is, the corresponding log file is matched according to the monitoring directory, source file format, periodic monitoring start timestamp, and periodic monitoring end timestamp, and the first attribute information of the target log file is obtained.

[0078] S2: Determine the compression parameters of the target log file based on the first attribute information of the target log file.

[0079] It should be noted that compression parameters can refer to the compressed file name, compressed file format, size, etc.

[0080] In some specific implementations, determining the compression parameters of the target log file based on the first attribute information of the target log file includes:

[0081] Obtain the scheduling flag of the target log file to determine whether the real-time monitoring program is being started for the first time;

[0082] In response to the detection that the real-time monitoring program is being started for the first time, based on the start timestamp and end timestamp of the time period of the target log file, the target log file is saved to the first list, and the flag bit corresponding to the scheduling flag is cleared. Here, the flag bit generally refers to the first flag bit. Clearing the flag bit indicates that the first scheduling has been completed. The first list refers to the list of files within the timestamp range.

[0083] In response to the detection that the real-time monitoring program is not being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file;

[0084] Based on a preset mapping table and a mode allocation strategy, the target log files in the first list are categorized and saved to a second list. The second list refers to a temporary file list, including a temporary category file list and a priority file list. The relevant modes in the mode allocation strategy include at least one of the following: basic mode, normal priority mode, and category mode. The preset mapping table includes at least one of the following: mapping name and priority code. The mapping name corresponding to the basic mode is the creation time. The mapping names corresponding to the normal priority mode are ERROR, WARNNING, INFO, DEBUG, and OTHER. The mapping name of the category mode is the component name. The priority code range of the normal priority mode is 0-4. The priority code range of the category mode can be expanded according to the number of sub-modules.

[0085] Based on the first attribute information of the target log file in the second list, the corresponding external data object compressed file is matched to determine the compression parameters of the target log file.

[0086] In some specific implementations, classifying and saving the target log files in the first list to the second list based on a preset mapping table and a pattern allocation strategy includes:

[0087] Based on the interface identifier to which the target log file belongs, the mapping name corresponding to the target log file is determined. Specifically, based on the relevant identifier of the component, the corresponding mapping name can be matched from the preset mapping table.

[0088] Based on the mapping name, determine the priority code of the target log file;

[0089] The mode to which the target log file belongs is determined based on the priority code;

[0090] Based on the pattern of the target log files, multiple target log files are classified, and according to the classification results, the target log files are saved to a second list. Specifically:

[0091] When the target log file belongs to the basic mode, the file list within its corresponding timestamp range is defined as the second list;

[0092] When the target log file belongs to the normal priority mode, the target log file is classified according to the priority code of the target log file, and a priority file list is generated;

[0093] When the target log file belongs to the classification mode, the target log file is classified according to the name in the preset mapping table, and a temporary classification file list is generated;

[0094] Furthermore, when the target log file belongs to the basic mode, the log files in the file list within the timestamp range are used as compression objects;

[0095] When the target log file belongs to the normal priority mode, the log files in the priority file list are used as compression objects, and different priority files are defined as different compression objects;

[0096] When the target log file belongs to the classification mode, the log files in the temporary classification file list are used as compression objects, and different priority files are defined as different compression objects;

[0097] Based on the compression object, the compression parameters of the target log file are determined, including: determining the format of the compressed file according to its corresponding external data object compressed file, determining the file name of the compressed file by combining its corresponding mapping name, timestamp, and compressed file name, and determining the size of the compressed file according to the external data object compressed file.

[0098] S3: Based on the compression parameters, compress the target log file to obtain the target compressed file.

[0099] It should be noted that before compressing the target log file to obtain the target compressed file, the method further includes:

[0100] Based on the first size limit, determine whether the target log file meets the preset standard, that is, compare whether the size of the compressed file is less than the maximum size limit of the compressed file. If it is less, it meets the preset standard; if it is not less, it does not meet the preset standard.

[0101] In response to the detection that the target log file meets the preset standard, the target log file is compressed to obtain the target compressed file, and the earliest creation time and the last creation time of the log file are obtained;

[0102] If the target log file is detected to not meet the preset standard, the compressed log file is closed and renamed based on the earliest and last log file creation times (ensuring that the timestamp range of the log file within the compressed file can be accurately obtained through the compressed file name). The compressed file is also named based on the current time and the end time of the time period. At the same time, the new compressed file is created according to the compressed file format in the external data object, and the log file compression continues to obtain the target compressed file.

[0103] S4: Based on the target compressed file, synchronously update the stored data in the memory.

[0104] It should be noted that this step is as follows:

[0105] The required space for the target compressed file is the size of the compressed file.

[0106] Based on the required space occupancy, the memory is cleaned up, including:

[0107] In response to detecting that the target log file belongs to the basic mode, the storage is cleaned up according to the creation time order of the log files. That is, when the current remaining space of the storage is insufficient to store the target compressed file, the storage is cleaned up according to the creation time of the log files from oldest to newest.

[0108] In response to the detection that the target log file belongs to the normal priority mode or the classification mode, the memory is cleaned up according to the priority code of the log file. That is, according to the priority code 0-4 (ERROR / WARNNING / INFO / DEBUG / OTHER), based on the rule of descending priority, the log files with higher priority are cleaned up first. For example, log files corresponding to the OTHER level are cleaned up first, followed by log files corresponding to the DEBUG level, and so on.

[0109] To reduce frequent disk deletion and avoid resource waste, within a preset number of consecutive time periods (where the preset number can be set according to actual needs, such as 3 times), and the required space occupancy is greater than a preset memory size (referring to the current remaining memory space or a first limit size), the space to be cleaned is calculated using a preset algorithm. This preset algorithm includes:

[0110]

[0111] Where X represents the size of the space to be cleared, Y represents the first limit size, and N represents the cumulative value corresponding to the consecutive preset number of times. That is, when the required space occupancy is greater than the preset size of the memory within the consecutive preset number of time periods, the count starts from 1 and is accumulated. For example, if the preset number is 3, the cumulative value is 1; if the preset number is 4, the cumulative value is 2, and so on.

[0112] Based on the value of the space to be cleaned, the memory is cleaned;

[0113] The target compressed file is saved to the cleaned storage space to synchronously update the stored data in the storage space. The stored data in the storage space refers to all compressed files and the file content corresponding to the compressed files, such as the compressed file name.

[0114] The external module interface in the external data object can periodically provide feedback on component log file monitoring and compressed file information. At this time, the compressed file can be transmitted to the external controller or the cloud via the controller-to-controller Ethernet communication interface. Figure 2 As shown, it can include automotive T-BOX, autonomous driving controller, body controller, cockpit controller and other component controllers, which can be exported by maintenance personnel via a local USB flash drive or a remote cloud drive (such as a vehicle cloud server).

[0115] The above-mentioned log monitoring and compression method includes: monitoring and acquiring a target log file and its first attribute information according to a preset time period; determining compression parameters for the target log file based on the first attribute information; compressing the target log file based on the compression parameters to obtain a target compressed file; and synchronously updating the stored data in the memory based on the target compressed file. This application can realize periodic monitoring and compression of log files corresponding to any component, and simultaneously perform periodic cleaning of the memory based on the compressed file, thereby improving memory utilization. While avoiding resource waste, it facilitates developers or maintenance personnel to quickly analyze and solve problems, thereby reducing the time and economic costs of vehicle development.

[0116] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0117] Example 2: In one example, as Figure 3 As shown, a log monitoring and compression device is provided, including: a monitoring acquisition module, a compression parameter determination module, a compression module, and a storage data update module, wherein:

[0118] The monitoring and acquisition module is used to monitor and acquire the target log file and the first attribute information of the target log file according to a preset time period.

[0119] The compression parameter determination module is used to determine the compression parameters of the target log file based on the first attribute information of the target log file;

[0120] A compression module is used to compress the target log file based on the compression parameters to obtain a target compressed file;

[0121] The storage data update module is used to synchronously update the stored data in the memory based on the target compressed file.

[0122] In a preferred embodiment of the present invention, the device further includes a generation module, which is specifically used for:

[0123] Based on the data upload interface and the monitoring directory, the relevant parameters of the target compressed file are determined. The relevant parameters include at least one of the following: preset time period, source file format, compressed file format, compressed file name, and interface identifier.

[0124] Based on the second attribute information of the storage, a first limit size and a second limit size of the compressed file are determined. The second attribute information includes the storage capacity. The first limit size is the maximum size of the compressed file, and the second limit size is the maximum storage size occupied by the compressed file.

[0125] Based on the relevant parameters, the first limit size, and the second limit size, an external data object compressed file is generated for monitoring and compressing the target log file.

[0126] In a preferred embodiment of the present invention, the monitoring and acquisition module is specifically used for:

[0127] In response to the detection of the start of a preset time period, the log files uploaded by the target upload interface within the preset time period are continuously monitored based on a real-time monitoring program.

[0128] In response to the detection that a single preset time period has ended, all log files within the preset time period are retrieved;

[0129] Based on the compressed file of the external data object, the target log file is selected from all log files, and the first attribute information of the target log file is obtained. The first attribute information includes at least one of the following: file format, size, interface identifier, scheduling flag, time period start timestamp, and time period end timestamp.

[0130] In a preferred embodiment of the present invention, the compression parameter determination module is specifically used for:

[0131] Obtain the scheduling flag of the target log file to determine whether the real-time monitoring program is being started for the first time;

[0132] In response to the detection that the real-time monitoring program is being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file, and the flag bit corresponding to the scheduling flag is cleared.

[0133] In response to the detection that the real-time monitoring program is not being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file;

[0134] Based on a preset mapping table and a mode allocation strategy, the target log files in the first list are classified and saved to the second list. The preset mapping table includes at least one of the following: mapping name and priority code.

[0135] Based on the first attribute information of the target log file in the second list, the corresponding external data object compressed file is matched to determine the compression parameters of the target log file.

[0136] In a preferred embodiment of the present invention, the compression parameter determination module is further configured to:

[0137] Based on the interface identifier to which the target log file belongs, determine the mapping name corresponding to the target log file;

[0138] Based on the mapping name, determine the priority code of the target log file;

[0139] Based on the priority code, the mode to which the target log file belongs is determined, and the mode includes at least one of the following: basic mode, normal priority mode, and category mode;

[0140] Based on the pattern to which the target log files belong, multiple target log files are classified, and the target log files are saved to a second list according to the classification results.

[0141] In a preferred embodiment of the present invention, the device further includes a judgment module, which is specifically used for:

[0142] Based on the first size limit, determine whether the target log file meets the preset standard;

[0143] In response to the detection that the target log file meets a preset standard, the target log file is compressed to obtain a target compressed file.

[0144] In a preferred embodiment of the present invention, the stored data update module is specifically used for:

[0145] Obtain the required space usage of the target compressed file;

[0146] Based on the required space occupancy, the memory is cleaned up, including:

[0147] In response to detecting that the target log file belongs to the basic mode, the storage is cleaned up according to the creation time order of the log files;

[0148] In response to detecting that the target log file belongs to a normal priority mode or a category mode, the memory is cleaned up according to the priority code of the log file;

[0149] In response to the detection that the required space occupancy exceeds a preset memory size for a consecutive preset number of time periods, the amount of space to be cleared is calculated using a preset algorithm, which includes:

[0150]

[0151] Where X represents the size of the space to be cleared, Y represents the first limit size, and N represents the cumulative value corresponding to the preset number of consecutive times;

[0152] Based on the value of the space to be cleaned, the memory is cleaned;

[0153] The target compressed file is saved to the cleaned storage space to synchronously update the stored data in the storage space.

[0154] Specific limitations regarding the log monitoring and compression device can be found in the limitations of the log monitoring and compression method described above, and will not be repeated here. Each module in the aforementioned log monitoring and compression device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0155] Example 3: In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows. Figure 4As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a log monitoring and compression method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0156] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0158] S1: Monitor and obtain the target log file and its first attribute information according to a preset time period;

[0159] S2: Determine the compression parameters of the target log file based on the first attribute information of the target log file;

[0160] S3: Based on the compression parameters, compress the target log file to obtain a target compressed file;

[0161] S4: Based on the target compressed file, synchronously update the stored data in the memory.

[0162] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0163] Based on the data upload interface and the monitoring directory, the relevant parameters of the target compressed file are determined. The relevant parameters include at least one of the following: preset time period, source file format, compressed file format, compressed file name, and interface identifier.

[0164] Based on the second attribute information of the storage, a first limit size and a second limit size of the compressed file are determined. The second attribute information includes the storage capacity. The first limit size is the maximum size of the compressed file, and the second limit size is the maximum storage size occupied by the compressed file.

[0165] Based on the relevant parameters, the first limit size, and the second limit size, an external data object compressed file is generated for monitoring and compressing the target log file.

[0166] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0167] In response to the detection of the start of a preset time period, the log files uploaded by the target upload interface within the preset time period are continuously monitored based on a real-time monitoring program.

[0168] In response to the detection that a single preset time period has ended, all log files within the preset time period are retrieved;

[0169] Based on the compressed file of the external data object, the target log file is selected from all log files, and the first attribute information of the target log file is obtained. The first attribute information includes at least one of the following: file format, size, interface identifier, scheduling flag, time period start timestamp, and time period end timestamp.

[0170] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0171] Obtain the scheduling flag of the target log file to determine whether the real-time monitoring program is being started for the first time;

[0172] In response to the detection that the real-time monitoring program is being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file, and the flag bit corresponding to the scheduling flag is cleared.

[0173] In response to the detection that the real-time monitoring program is not being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file;

[0174] Based on a preset mapping table and a mode allocation strategy, the target log files in the first list are classified and saved to the second list. The preset mapping table includes at least one of the following: mapping name and priority code.

[0175] Based on the first attribute information of the target log file in the second list, the corresponding external data object compressed file is matched to determine the compression parameters of the target log file.

[0176] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0177] Based on the interface identifier to which the target log file belongs, determine the mapping name corresponding to the target log file;

[0178] Based on the mapping name, determine the priority code of the target log file;

[0179] Based on the priority code, the mode to which the target log file belongs is determined, and the mode includes at least one of the following: basic mode, normal priority mode, and category mode;

[0180] Based on the pattern to which the target log files belong, multiple target log files are classified, and the target log files are saved to a second list according to the classification results.

[0181] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0182] Based on the first size limit, determine whether the target log file meets the preset standard;

[0183] In response to the detection that the target log file meets a preset standard, the target log file is compressed to obtain a target compressed file.

[0184] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0185] Obtain the required space usage of the target compressed file;

[0186] Based on the required space occupancy, the memory is cleaned up, including:

[0187] In response to detecting that the target log file belongs to the basic mode, the storage is cleaned up according to the creation time order of the log files;

[0188] In response to detecting that the target log file belongs to a normal priority mode or a category mode, the memory is cleaned up according to the priority code of the log file;

[0189] In response to the detection that the required space occupancy exceeds a preset memory size for a consecutive preset number of time periods, the amount of space to be cleared is calculated using a preset algorithm, which includes:

[0190]

[0191] Where X represents the size of the space to be cleared, Y represents the first limit size, and N represents the cumulative value corresponding to the preset number of consecutive times;

[0192] Based on the value of the space to be cleaned, the memory is cleaned.

[0193] The target compressed file is saved to the cleaned storage space to synchronously update the stored data in the storage space.

[0194] Example 4: In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the following steps:

[0195] S1: Monitor and obtain the target log file and its first attribute information according to a preset time period;

[0196] S2: Determine the compression parameters of the target log file based on the first attribute information of the target log file;

[0197] S3: Based on the compression parameters, compress the target log file to obtain a target compressed file;

[0198] S4: Based on the target compressed file, synchronously update the stored data in the memory.

[0199] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0200] Based on the data upload interface and the monitoring directory, the relevant parameters of the target compressed file are determined. The relevant parameters include at least one of the following: preset time period, source file format, compressed file format, compressed file name, and interface identifier.

[0201] Based on the second attribute information of the storage, a first limit size and a second limit size of the compressed file are determined. The second attribute information includes the storage capacity. The first limit size is the maximum size of the compressed file, and the second limit size is the maximum storage size occupied by the compressed file.

[0202] Based on the relevant parameters, the first limit size, and the second limit size, an external data object compressed file is generated for monitoring and compressing the target log file.

[0203] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0204] In response to the detection of the start of a preset time period, the log files uploaded by the target upload interface within the preset time period are continuously monitored based on a real-time monitoring program;

[0205] In response to the detection that a single preset time period has ended, all log files within the preset time period are retrieved;

[0206] Based on the compressed file of the external data object, the target log file is selected from all log files, and the first attribute information of the target log file is obtained. The first attribute information includes at least one of the following: file format, size, interface identifier, scheduling flag, time period start timestamp, and time period end timestamp.

[0207] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0208] Obtain the scheduling flag of the target log file to determine whether the real-time monitoring program is being started for the first time;

[0209] In response to the detection that the real-time monitoring program is being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file, and the flag bit corresponding to the scheduling flag is cleared.

[0210] In response to the detection that the real-time monitoring program is not being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file;

[0211] Based on a preset mapping table and a mode allocation strategy, the target log files in the first list are classified and saved to the second list. The preset mapping table includes at least one of the following: mapping name and priority code.

[0212] Based on the first attribute information of the target log file in the second list, the corresponding external data object compressed file is matched to determine the compression parameters of the target log file.

[0213] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0214] Based on the interface identifier to which the target log file belongs, determine the mapping name corresponding to the target log file;

[0215] Based on the mapping name, determine the priority code of the target log file;

[0216] Based on the priority code, the mode to which the target log file belongs is determined, and the mode includes at least one of the following: basic mode, normal priority mode, and category mode;

[0217] Based on the pattern to which the target log files belong, multiple target log files are classified, and the target log files are saved to a second list according to the classification results.

[0218] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0219] Based on the first size limit, determine whether the target log file meets the preset standard;

[0220] In response to the detection that the target log file meets a preset standard, the target log file is compressed to obtain a target compressed file.

[0221] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0222] Obtain the required space usage of the target compressed file;

[0223] Based on the required space occupancy, the memory is cleaned up, including:

[0224] In response to detecting that the target log file belongs to the basic mode, the storage is cleaned up according to the creation time order of the log files;

[0225] In response to detecting that the target log file belongs to a normal priority mode or a category mode, the memory is cleaned up according to the priority code of the log file;

[0226] In response to the detection that the required space occupancy exceeds a preset memory size for a consecutive preset number of time periods, the amount of space to be cleared is calculated using a preset algorithm, which includes:

[0227]

[0228] Where X represents the size of the space to be cleared, Y represents the first limit size, and N represents the cumulative value corresponding to the preset number of consecutive times;

[0229] Based on the value of the space to be cleaned, the memory is cleaned;

[0230] The target compressed file is saved to the cleaned storage space to synchronously update the stored data in the storage space.

[0231] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0232] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0233] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A log monitoring and compression method, characterized in that, Applied to vehicle components, the method includes: According to a preset time period, monitor and obtain the target log file and the first attribute information of the target log file; Based on the first attribute information of the target log file, the compression parameters of the target log file are determined, wherein the first attribute information includes at least one of the following: time period start timestamp and time period end timestamp; Based on the compression parameters, the target log file is compressed to obtain a target compressed file; Based on the target compressed file, the stored data in the memory is updated synchronously; The step of determining the compression parameters of the target log file based on the first attribute information of the target log file includes: Based on the start and end timestamps of the time period of the target log file, the target log file is saved to the first list. Based on a preset mapping table and a pattern allocation strategy, the target log files in the first list are categorized and saved to the second list; Based on the first attribute information of the target log file in the second list, the corresponding external data object compressed file is matched to determine the compression parameters of the target log file; The preset mapping table includes at least one of the following: mapping name and priority code. The step of classifying and saving the target log files in the first list to the second list based on the preset mapping table and the mode allocation strategy includes: Based on the interface identifier to which the target log file belongs, determine the mapping name corresponding to the target log file; Based on the mapping name, determine the priority code of the target log file; Based on the priority code, the mode to which the target log file belongs is determined. The mode includes at least one of the following: basic mode, normal priority mode, and category mode. The mapping name of the category mode is the component name. Based on the pattern to which the target log files belong, multiple target log files are classified, and the target log files are saved to a second list according to the classification results; Wherein, when the target log file belongs to the basic mode, the file list within its corresponding timestamp range is defined as the second list, and the log files in the file list within the timestamp range are used as compression objects; When the target log file belongs to the normal priority mode, the target log file is classified according to the priority code of the target log file, and a priority file list is generated. The log files in the priority file list are used as compression objects, and different priority files are defined as different compression objects. When the target log file belongs to the classification mode, the target log file is classified according to the name in the preset mapping table, and a temporary classification file list is generated. The log files in the temporary classification file list are used as compression objects, and different priority files are defined as different compression objects. Based on the compression object, determine the compression parameters of the target log file.

2. The log monitoring and compression method according to claim 1, characterized in that, Before monitoring and acquiring the target log file and its first attribute information according to a preset time period, the method further includes: Based on the data upload interface and the monitoring directory, the relevant parameters of the target compressed file are determined. The relevant parameters include at least one of the following: preset time period, source file format, compressed file format, compressed file name, and interface identifier. Based on the second attribute information of the storage, a first limit size and a second limit size of the compressed file are determined. The second attribute information includes the storage capacity. The first limit size is the maximum size of the compressed file, and the second limit size is the maximum storage size occupied by the compressed file. Based on the relevant parameters, the first limit size, and the second limit size, an external data object compressed file is generated for monitoring and compressing the target log file.

3. The log monitoring and compression method according to claim 1, characterized in that, The step of monitoring and acquiring the target log file and its first attribute information according to a preset time period includes: In response to the detection of the start of a preset time period, the log files uploaded by the target upload interface within the preset time period are continuously monitored based on a real-time monitoring program. In response to the detection that a single preset time period has ended, all log files within the preset time period are retrieved; Based on the compressed file of the external data object, the target log file is selected from all log files, and the first attribute information of the target log file is obtained. The first attribute information includes at least one of the following: file format, size, interface identifier, and scheduling flag.

4. The log monitoring and compression method according to claim 3, characterized in that, The step of determining the compression parameters of the target log file based on the first attribute information of the target log file includes: Obtain the scheduling flag of the target log file to determine whether the real-time monitoring program is being started for the first time; In response to the detection that the real-time monitoring program is being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file, and the flag bit corresponding to the scheduling flag is cleared. In response to the detection that the real-time monitoring program is not being started for the first time, the target log file is saved to the first list based on the start timestamp and end timestamp of the time period of the target log file; Based on the preset mapping table and pattern allocation strategy, the target log files in the first list are classified and saved to the second list; Based on the first attribute information of the target log file in the second list, the corresponding external data object compressed file is matched to determine the compression parameters of the target log file.

5. The log monitoring and compression method according to claim 2, characterized in that, Before compressing the target log file to obtain the target compressed file, the method further includes: Based on the first size limit, determine whether the target log file meets the preset standard; In response to the detection that the target log file meets a preset standard, the target log file is compressed to obtain a target compressed file.

6. The log monitoring and compression method according to claim 4, characterized in that, The step of synchronously updating the stored data in the memory based on the target compressed file includes: Obtain the required space usage of the target compressed file; Based on the required space occupancy, the memory is cleaned up, including: In response to detecting that the target log file belongs to the basic mode, the storage is cleaned up according to the creation time order of the log files; In response to detecting that the target log file belongs to a normal priority mode or a category mode, the memory is cleaned up according to the priority code of the log file; In response to the detection that the required space occupancy exceeds a preset memory size for a consecutive preset number of time periods, the amount of space to be cleared is calculated using a preset algorithm, which includes: Where X represents the size of the space to be cleared, Y represents the first limit size, and N represents the cumulative value corresponding to the preset number of consecutive times; Based on the value of the space to be cleaned, the memory is cleaned; The target compressed file is saved to the cleaned storage space to synchronously update the stored data in the storage space.

7. A log monitoring and compression apparatus for implementing the log monitoring and compression method as described in claim 1, characterized in that, The device includes: The monitoring and acquisition module is used to monitor and acquire the target log file and the first attribute information of the target log file according to a preset time period. The compression parameter determination module is used to determine the compression parameters of the target log file based on the first attribute information of the target log file; A compression module is used to compress the target log file based on the compression parameters to obtain a target compressed file; The storage data update module is used to synchronously update the stored data in the memory based on the target compressed file.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images