Data management method and system for operation logs, electronic equipment and storage medium

Abstract

The invention relates to a data management method and system for operation logs, electronic equipment and a storage medium. The management method comprises the steps that the operation logs for conducting the same operation under the same system module are obtained; the data format of the operation log is a key value pair format; generating a redundancy template based on the same part of each operation log, and generating difference fragment data corresponding to the operation logs according to different parts of each operation log; the redundant template comprises parts with the same values in the operation logs; and combining the redundant template and the difference fragment data corresponding to each operation log to generate a compressed file. According to the scheme, the operation logs for performing the same operation under the same system module are compressed, the operation mode in the part of the operation logs is more fixed, and the same part of the operation logs is higher, so that the same part of the operation logs is determined to generate the redundant template; and finally, generating a compressed file by combining the redundant template and the different fragment data.

Description

Technical Field

[0001] The present invention belongs to the technical field of data compression, and in particular, relates to a data management method, system, electronic device and storage medium for an operation log. Background Art

[0002] The JSON data format has good readability and rich data types. It is widely used in structured data storage, such as archiving system operation logs: exporting operation log table records into JSON data, storing them in JSON files, and compressing them for storage. Each line of a JSON archive file is a JSON object, representing an operation log, and the structures of multiple JSON objects are consistent, so a JSON archive file can be considered a homogeneous collection.

[0003] As the system version iterations and project development time increase, the size of the configured JSON files is also increasing. When large JSON files need to be compressed, the main thread or system service efficiency will be seriously affected due to the large amount of resources consumed in compressing JSON files. Summary of the invention

[0004] To solve the above problems, the present disclosure provides a data management method, system, electronic device and storage medium for operation logs. The solution has a high compression ratio and low performance consumption. It can effectively reduce the disk space occupied by the operation log archive files in the scenario described in the invention and save storage costs.

[0005] In order to solve the above technical problems, the first aspect of the present invention provides a data management method for operation logs, the method comprising:

[0006] Obtaining an operation log of the same operation performed under the same system module; the data format of the operation log is a key-value pair format;

[0007] Generate a redundant template based on the same parts between the operation logs, and generate difference fragment data of the corresponding operation logs according to the different parts between the operation logs; the redundant template includes: the same parts between the values ​​in the operation logs;

[0008] The redundant template and the difference segment data corresponding to each of the operation logs are combined to generate a compressed file.

[0009] According to a preferred implementation of the present invention, combining the redundant template and the difference fragment data corresponding to each of the operation logs to generate a compressed file includes:

[0010] Read each of the operation logs in sequence to determine whether a temporary compressed file corresponding to each of the operation logs exists;

[0011] If the temporary compressed file does not exist, a temporary compressed file corresponding to each of the operation logs is generated, the redundant template is written into the first line of the temporary compressed file, and the difference segment data corresponding to the currently read operation log is added to the end of the temporary compressed file;

[0012] If the temporary compressed file exists, adding the difference segment data corresponding to the currently read operation log to the end of the temporary compressed file;

[0013] After each of the operation logs is read, the temporary compressed file is obtained as the compressed file.

[0014] According to a preferred embodiment of the present invention, the generating of temporary compressed files corresponding to the respective operation logs comprises:

[0015] Obtaining log data corresponding to system modules and operations in the redundant template;

[0016] A temporary compressed file is generated, and a file name of the temporary compressed file is generated according to the log data corresponding to the system module and the operation; the file name includes: the log data corresponding to the system module and the operation.

[0017] According to a preferred implementation manner of the present invention, reading each of the operation logs in sequence and determining whether a temporary compressed file corresponding to each of the operation logs exists includes:

[0018] Extracting the system modules and log data corresponding to the operations in the redundant templates of each of the operation logs to generate verification data;

[0019] It is determined through the verification data whether there is a temporary compressed file corresponding to each of the operation logs.

[0020] According to a preferred embodiment of the present invention, the management method further comprises:

[0021] Combine the operation logs belonging to the same system module and the same operation to generate an operation log group;

[0022] After the compressed files are generated based on the operation log groups, the compressed files are merged to obtain a complete compressed file, wherein a separator is present between the compressed files in the complete compressed file.

[0023] According to a preferred embodiment of the present invention, the generating of a redundant template based on the same parts between the operation logs includes:

[0024] Comparing the operation logs, determining the same parts between the operation logs, and generating redundant templates for the same parts;

[0025] The management method further comprises: traversing each of the operation logs through the redundant template, and adding a difference mark to a portion of each of the operation logs that is inconsistent with the redundant template;

[0026] The generating of the difference fragment data of the corresponding operation log according to the different parts between the operation logs includes:

[0027] Each of the operation logs is read, and the content corresponding to the difference identifier in the operation log is extracted to obtain the difference segment data.

[0028] According to a preferred embodiment of the present invention, the management method further comprises:

[0029] Reading the compressed file, and identifying redundant templates in the compressed file;

[0030] Traversing each of the difference fragment data corresponding to the redundant template, replacing the variables in the redundant template one by one with the elements in the difference fragment data, and obtaining a complete decompression operation log when all the variables in the redundant template are replaced, and adding it to the decompressed file;

[0031] After decompression operation logs are generated corresponding to all the difference segment data, the decompression of the compressed file is completed to obtain a decompressed file.

[0032] In order to solve the above technical problem, the second aspect of the present invention proposes a data management system for operation logs, the management system comprising:

[0033] A log acquisition module is used to acquire operation logs of the same operation performed under the same system module; the data format of the operation logs is a key-value pair format;

[0034] A log processing module, used to generate a redundant template based on the same parts between the operation logs, and generate difference fragment data of the corresponding operation logs according to the different parts between the operation logs; the redundant template includes: the same parts between the values ​​in the operation logs;

[0035] The log compression module is used to combine the redundant template and the difference fragment data corresponding to each of the operation logs to generate a compressed file.

[0036] In order to solve the above technical problem, the third aspect of the present invention provides an electronic device, comprising:

[0037] Processor; and

[0038] A memory storing computer executable instructions, wherein when the computer executable instructions are executed, the processor executes the method described in any one of the above embodiments.

[0039] In order to solve the above technical problems, the fourth aspect of the present invention proposes a computer storage medium, wherein the computer storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method described in any one of the above embodiments is implemented.

[0040] Compared with the prior art, the present invention has the following advantages: the present invention collects operation logs of the same operation from the same system module, creates a redundant template based on the same parts of the operation logs, the redundant template includes the same parts between the values ​​in the operation logs, and extracts the different parts as difference fragment data. Finally, the redundant template and the difference fragment data corresponding to each operation log are combined to generate a compressed file. This solution compresses the operation logs of the same operation under the same system module. The operation mode in this part of the operation log is more fixed, and the same parts between the operation logs are higher. The same parts between the operation logs are determined to generate a redundant template, and finally a compressed file is generated by combining the redundant template and each difference fragment data. This solution has a higher compression ratio and lower performance consumption. It can effectively reduce the disk space occupied by the operation log archive file in the scenario described in the invention and save storage costs.

[0041] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A first flow chart of a method for managing data of an operation log according to an embodiment of the present disclosure is shown;

[0044] Figure 2 A second flow chart of a method for managing data of an operation log according to an embodiment of the present disclosure is shown;

[0045] Figure 3A schematic diagram of a method flow for determining whether a temporary compressed file corresponding to each operation log exists according to an embodiment of the present disclosure is shown;

[0046] Figure 4 A third flow chart of a method for managing data of an operation log according to an embodiment of the present disclosure is shown;

[0047] Figure 5 A block diagram of a data management system for an operation log according to an embodiment of the present disclosure is shown;

[0048] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0050] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be referred to as the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0051] See also Figure 1 , Figure 1 This is a flow chart of a data management method for an operation log provided by the present invention. Figure 1 As shown, the method includes:

[0052] S11. Obtain an operation log of the same operation performed under the same system module; the data format of the operation log is a key-value pair format.

[0053] In this embodiment, the operation logs of the same operation performed under the same system module are processed. Such operation logs have more duplication. For example, when creating a new document in a certain folder, in addition to the inconsistency of information such as user identity, the log information such as the content of the operation and the result of the operation will have more duplication. The following two operation logs in JSON format can be referred to;

[0054]

[0055]

[0056] In the operation log, timeStamp indicates the timestamp, clientIp indicates the client IP, userId indicates the user ID, module indicates the system module, uri indicates the URI of the system interface called by the operation, operation indicates the operation content, and result indicates the operation result. The yellow-marked contents in the above two operation logs are different parts, while the other parts are repeated parts. For the operation logs of the same operation under the same system module, the log information of the system module will be consistent, and the url of the system interface called by the operation will also be consistent, and there will be more overlaps in the operation content and operation result parts. For other types of operating systems, the operation logs of the same operation under the same system module using other storage formats will also have higher overlaps.

[0057] S12, generating a redundant template based on the same parts between the operation logs, and generating difference fragment data of the corresponding operation logs according to the different parts between the operation logs; the redundant template includes: the same parts between the values ​​in the operation logs.

[0058] In this embodiment, a redundant template is generated by using the same parts between the operation logs, and then the different parts between the operation logs are used to generate difference fragment data. In this solution, the difference fragment data is not coherent, and the different data can be combined together in an array to generate the difference fragment data, or the difference fragment data can be generated in a dictionary manner.

[0059] In this embodiment, the redundant template includes: the same parts between the values ​​in each operation log. Since the keys of the operation logs are generally fixed character segments, the redundant template in this scheme includes not only the keys in the operation logs, but also the same parts between the values ​​in the operation logs. The more data the redundant template contains, the smaller the data volume of the final compressed file will be, that is, the higher the compression rate will be.

[0060] In this embodiment, the operation logs may be compared to determine the same parts between the operation logs, and then a redundant template may be generated for the same parts.

[0061] In this embodiment, each operation log is read, and the content corresponding to the difference identifier in the operation log is extracted to obtain the difference segment data.

[0062] S13: Combine the redundant template and the difference segment data corresponding to each operation log to generate a compressed file.

[0063] In this embodiment, the final compressed file contains redundant templates and difference fragment data corresponding to each operation log, and the compressed file contains all important information. Since the type of processed data is determined in advance, the operation log is processed according to the system module and operation, which ensures a very high data compression rate.

[0064] In this embodiment, this scheme is equivalent to dividing the global operation log into different operation log groups according to system modules and operations and processing them separately, so as to improve the compression rate of the operation log in the "operation log group". Finally, each operation log group is compressed according to the steps of this scheme to obtain corresponding compressed files.

[0065] Specifically, the operation logs belonging to the same system module and the same operation are combined to generate an operation log group; after generating compressed files based on the operation log groups, the compressed files are merged to obtain a complete compressed file, and there are separators between the compressed files in the complete compressed file. In this embodiment, the compressed files are separated by blank lines, that is, blank lines are used as separators. Of course, other separators can also be used as separators, and this solution does not make special restrictions on this.

[0066] In this embodiment, in order to more quickly distinguish data in the operation log, the redundant template may be used to traverse each operation log, and a difference mark may be added to the part of each operation log that is inconsistent with the redundant template.

[0067] Specifically, a pair of difference identifiers "$[" and "$]" are introduced; for the same module and the same URI, the information system generates the corresponding operation log based on the template. There are placeholders in the template, and a placeholder consists of a pair of difference identifiers wrapping a variable; in the generated operation log, the difference fragments from the template are prompted with paired difference identifiers, and the operation log after adding the difference identifier is as follows:

[0068]

[0069]

[0070] In this embodiment, the yellow-marked content in the operation log is the difference mark. The final generated redundant template is as follows:

[0071]

[0072]

[0073] In this embodiment, the "$[paramX$]" in the above template is a placeholder, which consists of a pair of difference identifiers and the variable "paramX". Add the placeholder as marked in yellow above to the redundant template, and keep the placeholder in the redundant template to facilitate the subsequent decompression of the compressed file.

[0074] The present invention collects operation logs of the same operation from the same system module, creates a redundant template based on the same parts of the operation logs, the redundant template includes the same parts between the values ​​in the operation logs, and extracts the different parts as difference fragment data. Finally, the redundant template and the difference fragment data corresponding to each operation log are combined to generate a compressed file. This solution compresses the operation logs of the same operation under the same system module. The operation mode in this part of the operation log is more fixed, and the same parts between the operation logs are higher. The same parts between the operation logs are determined to generate a redundant template, and finally a compressed file is generated by combining the redundant template and each difference fragment data. This solution has a high compression ratio and low performance consumption. It can effectively reduce the disk space occupied by the operation log archive file in the scenario described in the invention and save storage costs.

[0075] See also Figure 2 , Figure 2 The second flow chart of the data management method of an operation log provided by the present invention is as follows: Figure 2 As shown, the method includes:

[0076] S21. Read each operation log in sequence to determine whether a temporary compressed file corresponding to each operation log exists.

[0077] In this embodiment, the operation logs in this solution are read sequentially, and the reading order can be determined according to the time sequence of the generation timestamps of the operation logs.

[0078] In this embodiment, the temporary compressed file of this solution corresponds to the operation logs that conform to the same system module and the same operation, that is, the operation logs that conform to the same system module and the same operation are compressed by the temporary compressed file.

[0079] S22a. If no temporary compressed file exists, a temporary compressed file corresponding to each operation log is generated, and a redundant template is written into the first line of the temporary compressed file, and the difference segment data corresponding to the currently read operation log is added to the end of the temporary compressed file.

[0080] In this embodiment, when reading the first operation log among the operation logs, if the temporary compressed file is not generated in advance, the temporary compressed file does not exist at this time, and the temporary compressed file is generated in real time, the redundant template is written into the temporary compressed file, and the difference fragment data corresponding to the currently read operation log is added to the temporary compressed file to complete the compression of the first operation log.

[0081] S22b: If a temporary compressed file exists, add the difference segment data corresponding to the currently read operation log to the end of the temporary compressed file.

[0082] In this embodiment, when a temporary compressed file exists, it means that the temporary compressed file may have been generated in advance, or the operation log currently being read is not the first operation log. At this time, the difference segment data corresponding to the currently read operation log is added to the temporary compressed file.

[0083] S23. After all operation logs are read, a temporary compressed file is obtained as a compressed file.

[0084] In this embodiment, after all operation logs are read, a compressed file is obtained.

[0085] In a specific embodiment, for global operation logs, operation logs belonging to the same system module and the same operation may be combined to generate operation log groups; redundant templates corresponding to each operation log group and difference fragment data corresponding to each operation log may be generated.

[0086] Generate corresponding temporary compressed files according to each operation log group in advance. At this time, you can directly read the global operation logs in sequence, and add the difference fragment data corresponding to the read operation logs to the corresponding temporary compressed files. In this way, multiple compressed files corresponding to each operation log group will be generated in the end.

[0087] In this embodiment, in order to distinguish each temporary compressed file or determine the ownership of the temporary compressed file, since the operation logs in this solution belong to the same system module and the same operation, or the same operation log group belongs to the same system module and the same operation, the temporary compressed file can be generated by the system module and the operation. Specifically, the log data corresponding to the system module and the operation in the redundant template is obtained; the temporary compressed file is generated, and the file name of the temporary compressed file is generated according to the log data corresponding to the system module and the operation; the file name includes: the log data corresponding to the system module and the operation. In this solution, the log data corresponding to the system module and the operation is used as the file name of the temporary compressed file, so that the existence of the temporary compressed file can be quickly determined by the file name.

[0088] like Figure 3As shown, in this embodiment, S21 determines whether a temporary compressed file corresponding to each operation log exists, including the following steps:

[0089] S31, extracting the system modules and log data corresponding to the operations in the redundant template of each operation log, and generating verification data.

[0090] S32. Determine whether there is a temporary compressed file corresponding to each operation log by verifying the data.

[0091] In this embodiment, in combination with the above embodiment, it is determined whether the temporary compressed file exists in the system through the system module in the redundant template and the log data corresponding to the operation.

[0092] In this embodiment, when the temporary compressed file is generated, the character strings corresponding to the system module in the operation log and the URI of the system interface called by the operation are merged to serve as the name of the temporary compressed file.

[0093] See also Figure 4 , Figure 4 The third is a flow chart of a data management method for an operation log provided by the present invention. Figure 4 As shown, the method comprises the following steps:

[0094] S41, reading a compressed file, and identifying redundant templates in the compressed file.

[0095] In this embodiment, the file content is read from top to bottom, each redundant template is identified, and subsequent steps are performed on the difference segment data below the redundant template.

[0096] S42, traversing each difference fragment data corresponding to the redundant template, replacing the variables in the redundant template one by one with the elements in the difference fragment data, and obtaining a complete decompression operation log when all the variables in the redundant template are replaced, and adding it to the decompressed file.

[0097] In this embodiment, the elements in the difference fragment data are traversed, and the variables in the redundant template are replaced one by one, that is, the placeholders in the redundant template are replaced, and a decompression operation log is obtained and added to the decompressed file.

[0098] S43. After all the difference segment data are correspondingly generated with decompression operation logs, the compressed file is decompressed to obtain a decompressed file.

[0099] In this embodiment, the above steps are repeated until the difference segment data under all redundant templates are restored into decompression operation logs and appended to the decompressed file, thereby obtaining the final decompressed file.

[0100] See also Figure 5 , Figure 5This is a block diagram of a data management system for an operation log provided by the present invention. Figure 5 As shown, the management system includes: a log acquisition module 11, a log processing module 12 and a log compression module 13.

[0101] In this embodiment, the log acquisition module 11 is used to acquire the operation log of the same operation performed in the same system module; the data format of the operation log is a key-value pair format.

[0102] In this embodiment, the log processing module 12 is used to generate a redundant template based on the same parts between each operation log, and generate difference fragment data of the corresponding operation log according to the different parts between each operation log; the redundant template includes: the same parts between the values ​​in each operation log.

[0103] In this embodiment, the log compression module 13 is used to combine the redundant template and the difference segment data corresponding to each operation log to generate a compressed file.

[0104] In this embodiment, the log compression module 13 is specifically used to read each operation log in turn to determine whether a temporary compressed file corresponding to each operation log exists; if no temporary compressed file exists, a temporary compressed file corresponding to each operation log is generated, and a redundant template is written into the first line of the temporary compressed file, and the difference fragment data corresponding to the currently read operation log is added to the end of the temporary compressed file; if a temporary compressed file exists, the difference fragment data corresponding to the currently read operation log is added to the end of the temporary compressed file; after each operation log is read, a temporary compressed file is obtained as a compressed file.

[0105] In this embodiment, the log compression module 13 is specifically used to obtain the log data corresponding to the system modules and operations in the redundant template; generate a temporary compressed file, and generate the file name of the temporary compressed file according to the log data corresponding to the system modules and operations; the file name includes: the log data corresponding to the system modules and operations.

[0106] In this embodiment, the log compression module 13 is specifically used to extract the system modules and log data corresponding to the operation in the redundant template of each operation log, generate verification data, and determine whether there is a temporary compressed file corresponding to each operation log through the verification data.

[0107] In this embodiment, the management system further includes: a log grouping module, which is used to combine operation logs belonging to the same system module and the same operation to generate an operation log group.

[0108] In this embodiment, the management system further includes: a file merging module, which is used to merge the compressed files to obtain a complete compressed file after generating the compressed files based on the operation log groups, and there are separators between the compressed files in the complete compressed file.

[0109] In this embodiment, the log processing module 12 is specifically used to compare various operation logs, determine the same parts between various operation logs, and generate redundant templates for the same parts.

[0110] In this embodiment, the log processing module 12 is specifically used to read each operation log, extract the content corresponding to the difference identifier in the operation log, and obtain the difference segment data.

[0111] In this embodiment, the management system further includes: a difference identification module, which is used to traverse each operation log through the redundant template and add a difference identification to the part of each operation log that is inconsistent with the redundant template.

[0112] In this embodiment, the management system also includes: a decompression module, which is used to read the compressed file and identify the redundant template in the compressed file; traverse each difference fragment data corresponding to the redundant template, and replace the variables in the redundant template one by one with the elements in the difference fragment data. When all the variables in the redundant template are replaced, a complete decompression operation log is obtained and added to the decompressed file; after the decompression operation logs are generated corresponding to all the difference fragment data, the decompression of the compressed file is completed to obtain the decompressed file.

[0113] like Figure 6 As shown, an embodiment of the present invention provides an electronic device, including a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140;

[0114] Memory 1130, used for storing computer programs;

[0115] The processor 1110 is used to implement any of the above management methods when executing the program stored in the memory 1130.

[0116] In the electronic device provided by the embodiment of the present invention, the processor 1110 obtains the operation log of the same operation performed under the same system module by executing the program stored in the memory 1130; the data format of the operation log is a key-value pair format; a redundant template is generated based on the same parts between each operation log, and the difference fragment data of the corresponding operation log is generated according to the different parts between each operation log; the redundant template includes: the same parts between the values ​​in each operation log; the redundant template and the difference fragment data corresponding to each operation log are combined to generate a compressed file.

[0117] The communication bus 1140 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0118] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0119] The memory 1130 may include a random access memory 1130 (RAM), or may include a non-volatile memory 1130 (non-volatile memory), such as at least one disk memory 1130. Optionally, the memory 1130 may also be at least one storage device located away from the processor 1110.

[0120] The above-mentioned processor 1110 can be a general-purpose processor 1110, including a central processing unit 1110 (CPU), a network processor 1110 (NP), etc.; it can also be a digital signal processor 1110 (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0121] An embodiment of the present invention provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors 1110 to implement the management method of any of the above embodiments.

[0122] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidStateDisk (SSD)), etc.

[0123] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A data management method for operation logs, characterized in that: The management method comprises: Obtaining an operation log of the same operation performed under the same system module; the data format of the operation log is a key-value pair format; Generate a redundant template based on the same parts between the operation logs, and generate difference fragment data of the corresponding operation logs according to the different parts between the operation logs; the redundant template includes: the same parts between the values ​​in the operation logs; The redundant template and the difference segment data corresponding to each of the operation logs are combined to generate a compressed file.

2. The management method according to claim 1, characterized in that: The step of combining the redundant template and the difference fragment data corresponding to each of the operation logs to generate a compressed file includes: Read each of the operation logs in sequence to determine whether a temporary compressed file corresponding to each of the operation logs exists; If the temporary compressed file does not exist, a temporary compressed file corresponding to each of the operation logs is generated, the redundant template is written into the first line of the temporary compressed file, and the difference segment data corresponding to the currently read operation log is added to the end of the temporary compressed file; If the temporary compressed file exists, adding the difference segment data corresponding to the currently read operation log to the end of the temporary compressed file; After each of the operation logs is read, the temporary compressed file is obtained as the compressed file.

3. The management method according to claim 2, characterized in that: The generating of temporary compressed files corresponding to the respective operation logs comprises: Obtaining log data corresponding to system modules and operations in the redundant template; A temporary compressed file is generated, and a file name of the temporary compressed file is generated according to the log data corresponding to the system module and the operation; the file name includes: the log data corresponding to the system module and the operation.

4. The management method according to claim 3, characterized in that: The step of sequentially reading each of the operation logs to determine whether a temporary compressed file corresponding to each of the operation logs exists includes: Extracting the system modules and log data corresponding to the operations in the redundant templates of each of the operation logs to generate verification data; It is determined through the verification data whether there is a temporary compressed file corresponding to each of the operation logs.

5. The management method according to claim 1, characterized in that: The management method also includes: Combine the operation logs belonging to the same system module and the same operation to generate an operation log group; After the compressed files are generated based on the operation log groups, the compressed files are merged to obtain a complete compressed file, wherein a separator is present between the compressed files in the complete compressed file.

6. The management method according to claim 1, characterized in that: The generating a redundant template based on the same parts between the operation logs includes: Comparing the operation logs, determining the same parts between the operation logs, and generating redundant templates for the same parts; The management method further comprises: traversing each of the operation logs through the redundant template, and adding a difference mark to a portion of each of the operation logs that is inconsistent with the redundant template; The generating of difference fragment data of corresponding operation logs according to different parts between the operation logs includes: Each of the operation logs is read, and the content corresponding to the difference identifier in the operation log is extracted to obtain the difference segment data.

7. The management method according to any one of claims 1 to 6, characterized in that: The management method also includes: Reading the compressed file, and identifying redundant templates in the compressed file; Traversing each of the difference fragment data corresponding to the redundant template, replacing the variables in the redundant template one by one with the elements in the difference fragment data, and obtaining a complete decompression operation log when all the variables in the redundant template are replaced, and adding it to the decompressed file; After decompression operation logs are generated corresponding to all the difference segment data, the decompression of the compressed file is completed to obtain a decompressed file.

8. A data management system for operation logs, characterized in that: The management system comprises: A log acquisition module is used to acquire operation logs of the same operation performed under the same system module; the data format of the operation logs is a key-value pair format; A log processing module, used to generate a redundant template based on the same parts between the operation logs, and generate difference fragment data of the corresponding operation logs according to the different parts between the operation logs; the redundant template includes: the same parts between the values ​​in the operation logs; The log compression module is used to combine the redundant template and the difference fragment data corresponding to each of the operation logs to generate a compressed file.

9. An electronic device, characterized in that: include: processor; as well as A memory storing computer executable instructions which, when executed, cause the processor to perform the method according to any one of claims 1-7.

10. A computer storage medium, characterized in that: in, The computer storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images