Log data compression processing method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses a log data compression processing method, device, electronic device and storage medium, the method comprising: obtaining each first attribute name in the first log data to be processed; determining the first compression name corresponding to each first attribute name according to the mapping relationship between the currently stored attribute name and the compression name and each first attribute name in the first log data, wherein the number of characters in the first compression name is less than the number of characters in the first attribute name; replacing each first attribute name in the first log data with the corresponding first compression name, and obtaining the second log data after compression processing. Through the technical solution of the embodiment of the present invention, the amount of log data can be greatly reduced without destroying the text structure of the log data.

Description

Technical Field

[0001] The embodiments of the present invention relate to Internet technology, and in particular to a log data compression processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of Internet technology, the system service architecture has evolved from the original monolithic to distributed and currently the most popular microservice style, making the request interaction process between services more complicated.

[0003] Usually, the server can write the generated log files into a full-text search engine (for example, Elasticsearch search server), so that when a fault occurs, the log keywords can be used to search the logs in the full-text search engine, so that the cause of the fault can be quickly analyzed based on the search results. As the number of application visits increases, the amount of log data generated also increases, so the log data needs to be compressed to reduce the amount of log data.

[0004] Currently, log data is usually compressed using a common data compression method, and then the compressed log data is written into a log file or a full-text search engine.

[0005] However, in the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0006] Commonly used data compression methods will make the compressed log text different from the original log text structure, thereby destroying the text structure of the log data, making it impossible for the full-text search engine to perform word segmentation and indexing, and then unable to perform full-text search based on keywords, reducing search efficiency. Summary of the invention

[0007] The embodiments of the present invention provide a log data compression processing method, device, electronic device and storage medium, so as to greatly reduce the amount of log data without destroying the text structure of the log data.

[0008] In a first aspect, an embodiment of the present invention provides a log data compression processing method, comprising:

[0009] Obtain each first attribute name in the first log data to be processed;

[0010] Determine, according to a mapping relationship between a currently stored attribute name and a compressed name and each first attribute name in the first log data, a first compressed name corresponding to each first attribute name, wherein the number of characters in the first compressed name is less than the number of characters in the first attribute name;

[0011] Each of the first attribute names in the first log data is replaced with the corresponding first compression name to obtain compressed second log data.

[0012] In a second aspect, an embodiment of the present invention further provides a log data compression processing device, comprising:

[0013] A first log data acquisition module, used to acquire each first attribute name in the first log data to be processed;

[0014] a first compressed name determining module, configured to determine a first compressed name corresponding to each first attribute name according to a mapping relationship between a currently stored attribute name and a compressed name and each first attribute name in the first log data, wherein the number of characters in the first compressed name is less than the number of characters in the first attribute name;

[0015] The first attribute name replacement module is used to replace each of the first attribute names in the first log data with the corresponding first compression name to obtain compressed second log data.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the log data compression processing method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, a log data compression processing method as provided in any embodiment of the present invention is implemented.

[0021] The embodiments of the above invention have the following advantages or beneficial effects:

[0022] By determining the first compressed name corresponding to each first attribute name in the first log data to be processed according to the mapping relationship between the currently stored attribute names and the compressed names, and replacing each first attribute name in the first log data with a corresponding first compressed name with a smaller number of characters, the amount of log data can be greatly reduced. Moreover, since only the first attribute name in the first log data is replaced and the rest of the parts remain unchanged, the text structure of the log data is not destroyed, so that full-text search can be performed based on keywords later, thereby improving search efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a log data compression processing method provided by Embodiment 1 of the present invention;

[0024] Figure 2 This is an example of log data compression involved in the first embodiment of the present invention;

[0025] Figure 3 is a flow chart of a log data compression processing method provided by Embodiment 2 of the present invention;

[0026] Figure 4 is a flow chart of a log data compression processing method provided by Embodiment 3 of the present invention;

[0027] Figure 5 This is an example of a log data compression processing architecture involved in Embodiment 3 of the present invention;

[0028] Figure 6 It is a structural schematic diagram of a log data compression processing device provided by Embodiment 4 of the present invention;

[0029] Figure 7 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a log data compression processing method provided in the first embodiment of the present invention. This embodiment is applicable to the case of compressing log data with a serialized string structure. The method can be executed by a log data compression processing device, which can be implemented by software and / or hardware and integrated into the application end corresponding to the application, wherein the application end can be a client with the application installed, or a server end serving the application. Figure 1 As shown, the method specifically comprises the following steps:

[0033] S110: Obtain each first attribute name in the first log data to be processed.

[0034] Among them, the first log data may refer to the log data with a string structure after serialization, or may refer to the log data to be serialized, that is, the request response data and / or program internal data in the application end to be serialized. That is to say, this embodiment may perform compression processing after serializing the log data, or may perform compression processing during the serialization of the log data. Serialization may refer to the process of converting the request response data and / or program internal data in the application end into a form that can be stored or transmitted. The string structure may refer to a data structure in the form of a string represented by a structured language. The structured language may be but is not limited to JSON (JavaScript Object Notation) or XML (Extensible Markup Language). The attribute name may refer to the name of the attribute field contained in the first log data.

[0035] Specifically, based on business needs, the request response data and / or program internal data generated by the application side can be stored as log data in a log file so that fault analysis can be performed later based on the log data. In the case where the log data is compressed after being serialized, when the application side generates the request response data and / or program internal data to be stored, it can use a structured language to serialize the request response data object and / or program internal data object, obtain the serialized request response data and / or program internal data with a string structure, that is, the first log data, and obtain the names of the first attributes contained in the first log data.

[0036] It should be noted that the data characteristics of the first log data to be processed are: there is a large amount of list data, and the attribute name field is fixed, so there will be a large number of repeated attribute name fields, resulting in a large amount of log data.

[0037] S120. Determine a first compressed name corresponding to each first attribute name according to a mapping relationship between currently stored attribute names and compressed names and each first attribute name in the first log data, wherein the number of characters in the first compressed name is less than the number of characters in the first attribute name.

[0038] Among them, the compressed name may refer to the name after the character compression of the attribute name. The characters used in the compressed name may refer to the provisions of the programming language identifier to ensure the validity of the compressed name. For example, the compressed name may be set using $, underscores, numbers, and uppercase and lowercase letters. The mapping relationship between the attribute name and the compressed name may refer to a one-to-one correspondence between the attribute name and the compressed name. The mapping relationship may be set in advance to improve the mapping efficiency; it may also be set in real time to reduce the memory space occupancy rate. The number of characters in each compressed name is less than the number of characters in the corresponding attribute name so as to reduce the number of characters in the attribute name. For example, the attribute name is: name, and its corresponding compressed name is: a. The compressed name in this embodiment may be composed of one or two characters to minimize the number of characters and improve the compression effect.

[0039] The mapping relationship between the attribute name and the compressed name can be saved in the form of a mapping table. The mapping table can be implemented using a hash table. The mapping relationship in this embodiment may include a forward mapping relationship (i.e., a forward mapping table) and a reverse mapping relationship (i.e., a reverse mapping table). The forward mapping relationship refers to the mapping from the attribute name to the compressed name. The reverse mapping relationship refers to the mapping from the compressed name to the attribute name. For example, if the attribute name is: message, and the corresponding compressed name is b, then the forward mapping relationship is message->b, and the reverse mapping relationship is b->message.

[0040] Specifically, the compressed name corresponding to each attribute name that may be used in the log data can be pre-set, a mapping relationship including all attribute names can be generated, and the generated mapping relationship can be stored, so that when the first log data is compressed, the first compressed name corresponding to each first attribute name in the first log data can be quickly and directly determined based on the stored mapping relationship. For example, based on the forward mapping relationship between the attribute name and the compressed name, that is, the forward mapping table, the first compressed name corresponding to each first attribute name can be determined more quickly, thereby improving the mapping efficiency.

[0041] S130: Replace each first attribute name in the first log data with a corresponding first compressed name to obtain compressed second log data.

[0042] Specifically, each first attribute name in the first log data is replaced with a corresponding first compressed name with fewer characters, thereby compressing all first attribute names of the first log data and reducing the amount of log data. Figure 2 An example of log data compression is given. Figure 2The left side of the figure is the first log data, and the right side is the compressed second log data. It can be seen that the amount of log data can be greatly reduced by compressing all the first attribute names. This embodiment only replaces the first attribute name in the first log data, and does not change the rest of the first log data, so as not to destroy the text structure of the first log data, so that the full-text log search can be performed based on keywords later, thereby improving the search efficiency.

[0043] It should be noted that there are often many repeated attribute names in the different object information generated by the application end, and the number of different attribute names will not be too large in general, usually there are hundreds to thousands of different attribute names, so by compressing the attribute names based on the mapping relationship, the overall log data volume can be greatly reduced. In addition, in different data application scenarios, the attribute value information corresponding to the same attribute name is different and unpredictable, so there is no need to compress the attribute value information corresponding to the first attribute name, so as to facilitate the subsequent use of keywords for full-text log search and improve search efficiency.

[0044] The technical solution of this embodiment can greatly reduce the amount of log data by determining the first compressed name corresponding to each first attribute name in the first log data to be processed according to the mapping relationship between the currently stored attribute name and the compressed name, and replacing each first attribute name in the first log data with a corresponding first compressed name with fewer characters. Moreover, since only the first attribute name in the first log data is replaced, the rest of the name remains unchanged, thereby not destroying the text structure of the log data, so that full-text search can be performed based on keywords later, thereby improving search efficiency.

[0045] Based on the above technical solution, S110 may include: obtaining the first log data to be serialized, and when serializing the first log data, parsing each first attribute name in the first log data. Correspondingly, "obtaining the compressed second log data" in S130 includes: performing serialization splicing processing based on each of the first compressed names to obtain the serialized second log data.

[0046] Specifically, in the case of performing compression processing during the serialization process of log data, the application side can directly use the generated request response data and / or program internal data to be stored as the first log data to be serialized, and parse each first attribute name in the first log data when serializing the first log data, so as to replace each first attribute name in the first log data with a corresponding first compressed name with a smaller number of characters, thereby greatly reducing the amount of log data. And, in accordance with the format splicing method required by the structured language, based on each first compressed name, the processed first log data is serialized and spliced ​​to obtain the serialized second log data. By directly compressing the log data during the serialization process, the amount of serialized log data can be greatly reduced, thereby reducing the memory space occupancy rate of the serialized log data, thereby improving the memory recovery efficiency of the application side and improving system performance.

[0047] Based on the above technical solution, S120 may include: detecting whether there is a compressed name corresponding to each first attribute name in the first log data in the mapping relationship between the currently stored attribute name and the compressed name; if so, determining the compressed name corresponding to the first attribute name as the first compressed name corresponding to the first attribute name; if not, determining the first compressed name corresponding to the first attribute name based on a preset mapping method, and storing the first attribute name and the corresponding first compressed name in the mapping relationship to update the mapping relationship.

[0048] The preset mapping method may be a pre-set method for establishing a mapping relationship between an attribute name and a compression name.

[0049] Specifically, this embodiment can determine the compressed name corresponding to each attribute name involved in real time during the process of compressing the attribute name, thereby establishing and storing the mapping relationship corresponding to each attribute name in real time, so that the currently stored mapping relationship only includes the mapping relationship corresponding to the currently involved attribute name, and does not include the mapping relationship corresponding to the attribute name not currently involved, thereby eliminating the need to store useless mapping relationships, reducing memory space occupancy and improving memory utilization.

[0050] Exemplarily, for each first attribute name in the first log data, it can be detected whether there is a mapping relationship corresponding to the first attribute name in the currently stored mapping relationship, such as the currently stored forward mapping table. If it is stored, the compressed name corresponding to the first attribute name can be directly determined based on the mapping relationship. If it does not exist, it indicates that the mapping relationship corresponding to the first attribute name needs to be generated in real time. At this time, the first compressed name corresponding to the first attribute name can be determined based on the preset mapping method. For example, various usable compressed names can be generated and stored in advance using preset characters, and a cursor pointer can be initialized. The cursor pointer is used to point to the storage position of a compressed name to be used, so that the storage position of the next compressed name to be used can be obtained based on the game pointer, and the compressed name stored in the storage position is determined as the first compressed name corresponding to the first attribute name. After determining the first compressed name, the cursor pointer can be moved back one position based on the compressed name usage order to point the cursor pointer to the storage position of the next compressed name to be used, so that the first compressed name can be quickly generated based on the cursor pointer, thereby improving the mapping efficiency. The order in which the compressed names are used may be generated based on the number of characters contained in the compressed names from small to large, so that the compressed names with the least number of characters are used first, further reducing the amount of log data and improving the compression effect.

[0051] By storing the mapping relationship between the first attribute name and the corresponding first compressed name, the currently stored mapping relationship is updated in real time. For example, the generated mapping relationship between the first attribute name and the corresponding first compressed name can be stored in the forward mapping table, so that the first compressed name corresponding to the first attribute name can be directly queried from the forward mapping table later, so that the currently stored mapping relationships are all mapping relationships corresponding to the first attribute names involved, and useless mapping relationships are not stored, thereby reducing memory space occupancy.

[0052] Embodiment 2

[0053] Figure 3 This is a flow chart of a log data compression processing method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment further optimizes "determining the first compression name corresponding to the first attribute name based on a preset mapping method". The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0054] See also Figure 3 The log data compression processing method provided in this embodiment specifically includes the following steps:

[0055] S310: Obtain each first attribute name in the first log data to be processed.

[0056] S320, detecting whether there is a compression name corresponding to each first attribute name in the first log data in the mapping relationship between the currently stored attribute name and the compression name, if so, proceeding to step S330, if not, proceeding to step S340.

[0057] S330. Determine the compressed name corresponding to the first attribute name as the first compressed name corresponding to the first attribute name, and proceed to step S380.

[0058] Specifically, when a mapping relationship corresponding to the first attribute name exists in the currently stored mapping relationship, the first compressed name corresponding to the first attribute name can be directly obtained based on the mapping relationship. When all first attribute names in the first log data have determined corresponding first compressed names, step S380 can be entered to perform a compression processing operation on the attribute name.

[0059] S340: Obtain the current number of compression names currently in use.

[0060] The current number of compressed names currently in use may refer to the number of compressed names used in all currently stored mapping relationships. The mapping relationship between the attribute name and the compressed name in this embodiment is a one-to-one correspondence, so that the current number of compressed names currently in use is equal to the number of attribute names currently involved. For example, the current number of compressed names currently in use may be the number of compressed names included in the currently stored mapping table, or the number of attribute names included in the currently stored mapping table.

[0061] Specifically, when the mapping relationship corresponding to the first attribute name does not exist in the currently stored mapping relationship, it indicates that the first compressed name corresponding to the first attribute name needs to be generated in real time. At this time, the current number of compressed names currently in use can be counted based on the currently stored mapping relationship. Exemplarily, this embodiment can set a global variable for storing the current number of compressed names currently in use in real time, so that the current number can be quickly obtained based on the global variable without counting each time, thereby improving mapping efficiency.

[0062] S350: If it is detected that the current number is less than the first preset number, a first compressed name corresponding to the first attribute name is generated using a preset character, and the process proceeds to step S370.

[0063] The first preset number may refer to the total number of compressed names that can be generated using a preset character. When a compressed name is generated using a preset character, the preset character used needs to comply with the provisions of the programming language identifier. For example, the preset character may be $ or uppercase and lowercase letters. When a compressed name is generated using $ or uppercase and lowercase letters, the total number of compressed names that can be generated is 53, that is, the first preset number is 53.

[0064] Specifically, it can be determined whether a compressed name can be generated using a preset character by detecting whether the current number is less than the first preset number. If it is detected that the current number is less than the first preset number, it indicates that there is still an unused compressed name generated using a preset character. At this time, the preset character can be continued to be used to generate the first compressed name corresponding to the first attribute name. For example, a compressed name can be randomly generated using a preset character, and it can be detected whether the compressed name has been used in the currently stored mapping relationship. If not, the compressed name can be determined as the first compressed name corresponding to the first attribute name; if so, another compressed name needs to be generated until a compressed name that is not currently in use is generated as the first compressed name.

[0065] Exemplarily, "using a preset character to generate a first compressed name corresponding to the first attribute name" in S350 may include: based on the mapping order of the first preset character, determining the target preset character corresponding to the first attribute name according to the current quantity as the corresponding first compressed name.

[0066] The first preset character mapping order may refer to a mapping order generated by using the first available preset characters. For example, the first available preset characters are $ and uppercase and lowercase letters, so the first preset character mapping order is obtained by sorting $ and uppercase and lowercase letters.

[0067] Specifically, various preset characters that can be used as the first position can be pre-stored based on the mapping order of the first preset character. When it is necessary to generate a compressed name using a preset character, the next preset character in the current number can be used as the target preset character according to the mapping order of the first preset character, and the target preset character can be used as the corresponding first compressed name. For example, if the current number is 5, it means that 5 preset characters have been used. At this time, the 6th preset character can be used as the corresponding first compressed name according to the mapping order of the first preset character, so that the first compressed name including only one preset character can be generated more quickly, thereby improving the mapping efficiency.

[0068] Exemplarily, based on the mapping order of the first preset character, determining the target preset character corresponding to the first attribute name according to the current quantity as the corresponding first compressed name may include: arranging the ASCII code values ​​corresponding to the first available preset character in ascending order, and determining the target ASCII code value corresponding to the first attribute name according to the current quantity; determining the target preset character corresponding to the target ASCII code value as the first compressed name corresponding to the first attribute name.

[0069] Specifically, the first available preset characters can be arranged in order from small to large according to the ASCII code value. For example, when the first available preset characters are $ and uppercase and lowercase letters, the sorted order is: $, uppercase letters A to Z, lowercase letters a to z. The target ASCII code value corresponding to the first attribute name can be determined according to the value of the current quantity. For example, if the current quantity X is 0, the target ASCII code value is 0; if the current quantity X is greater than 0 and less than or equal to 26, the target ASCII code value can be determined based on the formula 65+(X-1); if the current quantity X is greater than 26 and less than 53 (i.e., the first preset quantity), the target ASCII code value can be determined based on the formula 97+(X-27). By converting the target ASCII code value, the corresponding target preset character can be obtained, and the target preset character is used as the first compressed name corresponding to the first attribute name, so that the available target preset character can be determined in real time and quickly based on the ASCII code value, without pre-storing all the available preset characters in the first position, thereby reducing the memory space occupancy rate and improving the running performance.

[0070] It should be noted that after using a preset character to generate the first compressed name corresponding to the first attribute name, it indicates that a used compressed name has been added. At this time, the current number of currently used compressed names can be accumulated by 1, and the current number can be updated in real time so that the accurate current number can be directly obtained later, thereby improving mapping efficiency and mapping accuracy.

[0071] S360: If it is detected that the current number is greater than or equal to the first preset number and less than the second preset number, a first compressed name corresponding to the first attribute name is generated using two preset characters, and the process proceeds to step S370.

[0072] The second preset number may refer to the total number of compressed names that can be generated using two preset characters. When generating a compressed name using two preset characters, the two preset characters used need to comply with the provisions of the programming language identifier. For example, the first usable preset character may be $ or an uppercase or lowercase letter, and the second usable preset character may be $, an underscore, a number, or an uppercase or lowercase letter, so that the total number of compressed names that can be generated using any two preset characters is 3392, that is, the first preset number is 3392.

[0073] Specifically, the current number can be detected based on the first preset number and the second preset number to determine whether a compressed name can be generated using one preset character or two preset characters. If it is detected that the current number is greater than or equal to the first preset number, and less than the second preset number, it means that all compressed names generated using one preset character have been used, but there are currently unused compressed names generated using two preset characters. At this time, the two preset characters can be used to generate the first compressed name corresponding to the first attribute name. For example, a compressed name can be randomly generated using two available preset characters, and it can be detected whether the compressed name has been used in the currently stored mapping relationship. If not, the compressed name can be determined as the first compressed name corresponding to the first attribute name; if so, another compressed name needs to be generated until a compressed name that is not currently in use is generated as the first compressed name.

[0074] It should be noted that if it is detected that the current number is greater than or equal to the second preset number, and less than the total number of compressed names that can be generated using three preset characters, then the first compressed name corresponding to the first attribute name can be generated using three preset characters in a manner similar to that of generating compressed names using two preset characters, which will not be repeated here.

[0075] Exemplarily, "using two preset characters to generate a first compressed name corresponding to the first attribute name" in S360 may include: processing the current quantity according to the third preset quantity and the fourth preset quantity to determine a first reference value and a second reference value; based on the mapping order of the first preset character, determining the first preset character corresponding to the first attribute name according to the first reference value; based on the mapping order of the second preset character, determining the second preset character corresponding to the first attribute name according to the second reference value; and concatenating the first preset character and the second preset character to obtain a first compressed name corresponding to the first attribute name.

[0076] Among them, the third preset number may refer to the number of preset characters that can be used in the first position. The fourth preset number may refer to the number of preset characters that can be used in the second position. The third preset number in this embodiment may be equal to the first preset number. For example, when the preset character that can be used in the first position may be $ or uppercase and lowercase letters, the corresponding third preset number is 53. When the preset character that can be used in the second position may be $, underscore, number or uppercase and lowercase letters, the corresponding fourth preset number is 64. The first reference value may refer to a reference value used to determine the first preset character, such as the number of preset characters that have been used in the first position. The second reference value may refer to a reference value used to determine the second preset character, such as the number of preset characters that have been used in the second position. The second preset character mapping order, similar to the first preset character mapping order described above, may refer to a mapping usage order generated using each preset character that can be used in the second position. For example, each preset character that can be used in the second position is $, underscore, number and uppercase and lowercase letters, so that $, underscore, number and uppercase and lowercase letters are sorted to obtain the second preset character mapping order.

[0077] Specifically, the third preset number and the fourth preset number can be used to process the current number to determine the number of preset characters used in the first position (i.e., the first reference value) and the number of preset characters used in the second position (i.e., the second reference value). The first preset character corresponding to the first attribute name can be determined according to the first reference value based on the mapping order of the first preset character with reference to the above-described method. For example, the next preset character located at the first reference value can be used as the first preset character according to the mapping order of the first preset character; the target ASCII code value can also be determined according to the first reference value according to the arrangement order of the ASCII code values ​​corresponding to the first usable preset character from small to large, and the preset character corresponding to the target ASCII code value can be used as the first preset character. The determination method of the second preset character in this embodiment is similar to that of the first preset character. For example, the next preset character located at the second reference value can be used as the second preset character according to the mapping order of the second preset character; the target ASCII code value can also be determined according to the second reference value according to the arrangement order of the ASCII code values ​​corresponding to the second usable preset character from small to large, and the preset character corresponding to the target ASCII code value can be used as the second preset character. Exemplarily, the second available preset characters can be arranged in order from small to large according to the ASCII code value. For example, when the second available preset characters are $, underscore, numbers and uppercase and lowercase letters, the sorted order is: $, numbers 0 to 9, uppercase letters A to Z, underscore, lowercase letters a to z. The target ASCII code value corresponding to the second preset character can be determined according to the second reference value. For example, if the second reference value Y is 0, the target ASCII code value is determined to be 0; if the second reference value Y is greater than 0 and less than or equal to 10, the target ASCII code value can be determined based on formula 48+(Y-1); if the second reference value Y is greater than 10 and less than or equal to 36, the target ASCII code value can be determined based on formula 65+(Y-11). If the second reference value Y is equal to 37, the target ASCII code value is determined to be 95; if the second reference value Y is greater than 37 and less than 64 (i.e., the fourth preset number), the target ASCII code value can be determined based on formula 97+(Y-38). The corresponding second preset character can be obtained by converting the target ASCII code value. The first preset character and the second preset character are concatenated, and the concatenated result is the first compressed name corresponding to the first attribute name, so that the first compressed name including only two preset characters can be quickly generated, thereby improving the mapping efficiency.

[0078] Exemplarily, processing the current quantity according to the third preset quantity and the fourth preset quantity to determine the first reference value and the second reference value may include: determining the difference between the current quantity and the third preset quantity; dividing the difference by the fourth preset quantity, obtaining the quotient as the first reference value, and obtaining the remainder as the second reference value.

[0079] Specifically, the third preset number can be subtracted from the current number, and the difference can be divided by the fourth preset number, and the division result can be rounded down to obtain the first reference value, and the remainder of the division result can be obtained to obtain the second reference value.

[0080] It should be noted that after using two preset characters to generate the first compressed name corresponding to the first attribute name, it indicates that a used compressed name has been added. At this time, the current number of currently used compressed names can be accumulated by 1, and the current number can be updated in real time so that the accurate current number can be directly obtained later, thereby improving mapping efficiency and mapping accuracy.

[0081] S370: Store the first attribute name and the corresponding first compressed name in a mapping relationship to update the mapping relationship.

[0082] It should be noted that the operation of step S380 may be performed after the first compression name corresponding to each first attribute name in the first log data is determined by cyclically performing the operations of steps S350 to S370.

[0083] S380: Replace each first attribute name in the first log data with a corresponding first compressed name to obtain compressed second log data.

[0084] The technical solution of this embodiment generates a first compressed name corresponding to the first attribute name using one preset character when it is detected whether the current number of compressed names currently in use is less than a first preset number, and generates a first compressed name corresponding to the first attribute name using two preset characters when it is detected that the current number is greater than or equal to the first preset number and less than a second preset number, thereby preferentially generating a compressed name with the least number of characters, so as to further reduce the amount of compressed log data and further improve the compression effect.

[0085] Embodiment 3

[0086] Figure 4 This is a flow chart of a log data compression processing method provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment describes the log search process in detail after obtaining the second log data after compression processing. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0087] See also Figure 4 The log data compression processing method provided in this embodiment specifically includes the following steps:

[0088] S410: Obtain each first attribute name in the first log data to be processed.

[0089] Specifically, Figure 5 An example of a log data compression processing architecture is given. Figure 5 As shown, the log serialization module in the application end can serialize the request response data object and / or the program internal data object, obtain the request response data and / or the program internal data in the serialized string structure, that is, the first log data to be processed, and obtain each first attribute name in the first log data.

[0090] S420. Determine a first compressed name corresponding to each first attribute name according to a mapping relationship between currently stored attribute names and compressed names and each first attribute name in the first log data, wherein the number of characters in the first compressed name is less than the number of characters in the first attribute name.

[0091] Specifically, Figure 5 The centralized cache module in can be used to store the mapping relationship between the attribute name and the compression name, and provide the writing and query functions of the mapping relationship, so as to realize the mapping relationship data sharing between the application end and the full-text search engine. The local memory cache module in the application end can load the mapping relationship (such as the forward mapping table) from the centralized cache module to the local memory, so that the compression module can query the mapping relationship corresponding to each first attribute name in the first log data from the local memory cache module. If there is a mapping relationship, the compression module determines the corresponding first compression name based on the mapping relationship. If there is no mapping relationship, the compression module generates the corresponding first compression name in real time based on the preset mapping method, and saves the mapping relationship between the generated first compression name and the first attribute name in real time to the local memory cache module and the centralized cache module to realize the real-time update of the mapping relationship and ensure the accuracy of the mapping.

[0092] S430: Replace each first attribute name in the first log data with a corresponding first compressed name to obtain compressed second log data.

[0093] Specifically, Figure 5 As shown, after obtaining the serialized first log data, the log serialization module can obtain the first compression name (i.e., the compressed field name) corresponding to each first attribute name from the compression module, and replace each first attribute name with the corresponding first compression name, thereby obtaining the compressed second log data.

[0094] S440: Write the second log data into a log file, and write the log file into a full-text search engine, so as to perform a log search in the full-text search engine based on the log keywords input by the user, and obtain search result data.

[0095] The full-text search engine may refer to a server that retrieves log data that matches the log keyword input by the user from all log files and displays the search results to the user. For example, the full-text search engine may be, but is not limited to, an Elasticsearch search server. Log keywords may refer to any search terms input by the user based on business needs. For example, log keywords may be attribute names or attribute value information.

[0096] Specifically, Figure 5 As shown, the log serialization module in the application end can write the second log data into the log file, and output the log file to the search module of the full-text search engine for storage and indexing. The log file stored in the full-text search engine does not destroy the text structure of the original log data, so that the full-text search engine can perform word segmentation and indexing, and then can perform full-text search based on the input log keywords, thereby improving search efficiency.

[0097] Exemplarily, S440 may include: determining a search keyword for log search based on a log keyword input by a user; searching in each log file based on the search keyword to obtain third log data matching the search keyword; determining a third attribute name corresponding to each third compressed name based on a mapping relationship between currently stored attribute names and compression names and each third compressed name in the third log data; replacing each third compressed name of the third log data with a corresponding third attribute name to obtain decompressed search result data.

[0098] The search keyword may refer to a keyword that the full-text search engine can use to search for log data results. The third log data may refer to log search results that contain the search keyword and are directly searched by the full-text search engine.

[0099] Specifically, Figure 5As shown, the log keyword retrieval module in the full-text search engine can receive the log keyword input by the user. The local memory cache module in the full-text search engine can load the mapping relationship (such as the forward mapping table) from the centralized cache module to the local memory, so that the keyword compression module can determine the search keyword corresponding to the log keyword based on the stored mapping relationship. For example, if it is detected that the log keyword input by the user is an attribute name, the log keyword can be compressed based on the mapping relationship stored in the local memory cache module to obtain the search keyword after the log keyword is compressed. If it is detected that the log keyword input by the user is a non-attribute name, such as attribute value information, the log keyword can be directly determined as the search keyword without compression. The search module in the full-text search engine can search in all log files based on the search keyword to obtain the third log data matching the search keyword. The decompression module in the full-text search engine can query the mapping relationship from the local memory cache module, determine the third attribute name corresponding to each third compressed name in the third log data, and replace each third compressed name of the third log data with the corresponding third attribute name, obtain the search result data after decompression, and display the obtained search result data through the log display module, so that the fault analysis can be performed based on the displayed log data.

[0100] It should be noted that the mapping relationships loaded by the full-text search engine from the centralized cache module include mapping relationships corresponding to all the attribute names involved, so that the loaded mapping relationships will contain mapping relationships corresponding to log keywords and each third compressed name in the third log data, so that the corresponding search keywords and the corresponding third attribute names can be directly determined, thereby further improving the search efficiency by sharing the mapping relationships.

[0101] The technical solution of this embodiment, by writing the second log data into a log file, and writing the log file into a full-text search engine, ensures that the written log file does not destroy the text structure of the original log data, so that the full-text search engine can perform word segmentation and establish an index, and then perform a full-text search based on the input log keywords, thereby improving the search efficiency.

[0102] The following is an embodiment of a log data compression processing device provided by an embodiment of the present invention. The device and the log data compression processing methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the log data compression processing device, please refer to the embodiment of the above-mentioned log data compression processing method.

[0103] Embodiment 4

[0104] Figure 6This is a structural diagram of a log data compression processing device provided in Example 4 of the present invention. This embodiment is applicable to the situation where compression processing is performed on log data with a serialized string structure. The device specifically includes: a first attribute name acquisition module 610, a first compression name determination module 620 and a first attribute name replacement module 630.

[0105] Among them, the first attribute name acquisition module 610 is used to obtain each first attribute name in the first log data to be processed; the first compressed name determination module 620 is used to determine the first compressed name corresponding to each first attribute name according to the mapping relationship between the currently stored attribute name and the compressed name and each first attribute name in the first log data, wherein the number of characters in the first compressed name is less than the number of characters in the first attribute name; the first attribute name replacement module 630 is used to replace each first attribute name in the first log data with the corresponding first compressed name to obtain the second log data after compression processing.

[0106] Optionally, the first compressed name determining module 620 includes:

[0107] A mapping relationship detection unit, used to detect whether there is a compression name corresponding to each first attribute name in the first log data in the mapping relationship between the currently stored attribute name and the compression name;

[0108] a first determining unit, configured to determine the compressed name corresponding to the first attribute name as the first compressed name corresponding to the first attribute name;

[0109] The second determining unit is used to determine the first compressed name corresponding to the first attribute name based on a preset mapping method if not, and store the first attribute name and the corresponding first compressed name in a mapping relationship to update the mapping relationship.

[0110] Optionally, the second determining unit includes:

[0111] The current number acquisition subunit is used to acquire the current number of compression names currently in use;

[0112] A first generating subunit is used to generate a first compressed name corresponding to the first attribute name using a preset character if it is detected that the current number is less than a first preset number;

[0113] A second generating subunit is used to generate a first compressed name corresponding to the first attribute name using two preset characters if it is detected that the current number is greater than or equal to the first preset number and less than the second preset number;

[0114] The first preset number refers to the total number of compressed names that can be generated using one preset character; the second preset number refers to the total number of compressed names that can be generated using two preset characters.

[0115] Optionally, the first generating subunit is specifically used to: determine, based on the first preset character mapping order and according to the current quantity, a target preset character corresponding to the first attribute name as the corresponding first compressed name.

[0116] Optionally, the first generating sub-unit is further specifically used to: determine the target ASCII code value corresponding to the first attribute name according to the current quantity in order of the ASCII code values ​​corresponding to the first available preset character from small to large; and determine the target preset character corresponding to the target ASCII code value as the first compressed name corresponding to the first attribute name.

[0117] Optionally, the second generating sub-unit is specifically used to: process the current quantity according to a third preset quantity and a fourth preset quantity to determine a first reference value and a second reference value, wherein the third preset quantity refers to the number of first available preset characters, and the fourth preset quantity refers to the number of second available preset characters; based on the mapping order of the first preset character, determine the first preset character corresponding to the first attribute name according to the first reference value; based on the mapping order of the second preset character, determine the second preset character corresponding to the first attribute name according to the second reference value; concatenate the first preset character and the second preset character to obtain a first compressed name corresponding to the first attribute name.

[0118] Optionally, the second generating subunit is further specifically used to: determine the difference between the current quantity and the third preset quantity; divide the difference by the fourth preset quantity, and use the obtained quotient as the first reference value, and the obtained remainder as the second reference value.

[0119] Optionally, the first attribute name acquisition module 610 is specifically used to: obtain the first log data to be serialized, and when serializing the first log data, parse out each first attribute name in the first log data; the first attribute name replacement module 630 is also specifically used to: perform serialization and splicing processing based on each of the first compressed names to obtain the serialized second log data.

[0120] Optionally, the device further comprises:

[0121] The log writing module is used to write the second log data into a log file after obtaining the compressed second log data, and write the log file into a full-text search engine to perform log search in the full-text search engine based on log keywords input by the user to obtain search result data.

[0122] Optionally, the full-text search engine is specifically used to: determine the search keywords for log search based on the log keywords input by the user; search in each log file based on the search keywords to obtain third log data matching the search keywords; determine the third attribute name corresponding to each third compressed name based on the mapping relationship between the currently stored attribute name and the compression name and each third compressed name in the third log data; replace each third compressed name of the third log data with the corresponding third attribute name to obtain the search result data after decompression.

[0123] The log data compression processing device provided in the embodiment of the present invention can execute the log data compression processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the log data compression processing method.

[0124] It is worth noting that in the embodiment of the above-mentioned log data compression processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0125] Embodiment 5

[0126] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Figure 7 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 7 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0127] like Figure 7 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0128] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0129] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0130] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0131] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0132] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0133] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a log data compression processing method provided by the embodiment of the present invention, the method comprising:

[0134] Obtain each first attribute name in the first log data to be processed;

[0135] Determine, according to the mapping relationship between the currently stored attribute name and the compressed name and each first attribute name in the first log data, a first compressed name corresponding to each first attribute name, wherein the number of characters in the first compressed name is less than the number of characters in the first attribute name;

[0136] Each first attribute name in the first log data is replaced with a corresponding first compressed name to obtain compressed second log data.

[0137] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the log data compression processing method provided by any embodiment of the present invention.

[0138] Embodiment 6

[0139] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the log data compression processing method provided in any embodiment of the present invention are implemented. The method includes:

[0140] Obtain each first attribute name in the first log data to be processed;

[0141] Determine, according to the mapping relationship between the currently stored attribute name and the compressed name and each first attribute name in the first log data, a first compressed name corresponding to each first attribute name, wherein the number of characters in the first compressed name is less than the number of characters in the first attribute name;

[0142] Each first attribute name in the first log data is replaced with a corresponding first compressed name to obtain compressed second log data.

[0143] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0144] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0146] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0148] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A log data compression processing method, characterized in that: include: Obtain each first attribute name in the first log data to be processed; Detecting whether there is a compression name corresponding to each first attribute name in the first log data in the mapping relationship between the currently stored attribute name and the compression name; If not, get the current number of compressed names currently in use; If it is detected that the current number is less than the first preset number, a first compressed name corresponding to the first attribute name is generated using a preset character; If it is detected that the current number is greater than or equal to the first preset number and less than the second preset number, a first compressed name corresponding to the first attribute name is generated using two preset characters; The first attribute name and the corresponding first compressed name are stored in the mapping relationship to update the mapping relationship, wherein the first preset number refers to the total number of compressed names that can be generated using one preset character; the second preset number refers to the total number of compressed names that can be generated using two preset characters; Replacing each of the first attribute names in the first log data with the corresponding first compressed name to obtain compressed second log data; The second log data is written into a log file, and the log file is written into a full-text search engine, so as to segment and index the log file in the full-text search engine, and perform log search based on log keywords input by a user to obtain search result data, wherein the log keywords include attribute names.

2. The method according to claim 1, characterized in that The method further comprises: If so, the compressed name corresponding to the first attribute name is determined as the first compressed name corresponding to the first attribute name.

3. The method according to claim 1, characterized in that Using a preset character, generating a first compressed name corresponding to the first attribute name, including: Based on the first preset character mapping order, a target preset character corresponding to the first attribute name is determined according to the current number as the corresponding first compressed name.

4. The method according to claim 3, characterized in that Based on the first preset character mapping order, determining the target preset character corresponding to the first attribute name according to the current number as the corresponding first compressed name includes: According to the arrangement order of the ASCII code values ​​corresponding to the first available preset character from small to large, determine the target ASCII code value corresponding to the first attribute name according to the current number; The target preset character corresponding to the target ASCII code value is determined as the first compressed name corresponding to the first attribute name.

5. The method according to claim 1, characterized in that Using two preset characters, a first compressed name corresponding to the first attribute name is generated, including: Processing the current number according to a third preset number and a fourth preset number to determine a first reference value and a second reference value, wherein the third preset number refers to the number of preset characters that can be used in the first position, and the fourth preset number refers to the number of preset characters that can be used in the second position; Based on the first preset character mapping order, determine the first preset character corresponding to the first attribute name according to the first reference value; Based on the second preset character mapping order, determining the second preset character corresponding to the first attribute name according to the second reference value; The first preset character and the second preset character are concatenated to obtain a first compressed name corresponding to the first attribute name.

6. The method according to claim 5, characterized in that Processing the current number according to the third preset number and the fourth preset number to determine a first reference value and a second reference value includes: determining a difference between the current number and the third preset number; The difference is divided by a fourth preset number, the obtained quotient is used as the first reference value, and the obtained remainder is used as the second reference value.

7. The method according to claim 1, characterized in that Get each first attribute name in the first log data to be processed, including: Acquire first log data to be serialized, and when serializing the first log data, parse out each first attribute name in the first log data; The step of obtaining the compressed second log data includes: Serialization and concatenation processing is performed based on each of the first compressed names to obtain serialized second log data.

8. The method according to claim 1, characterized in that Perform log search based on the log keywords entered by the user to obtain search result data, including: Determine the search keyword for log search according to the log keyword input by the user; Based on the search keyword, searching in each of the log files to obtain third log data matching the search keyword; Determine, according to a mapping relationship between a currently stored attribute name and a compression name and each third compression name in the third log data, a third attribute name corresponding to each third compression name; Each of the third compressed names of the third log data is replaced with the corresponding third attribute name to obtain search result data after decompression.

9. A log data compression processing device, characterized in that: include: A first log data acquisition module, used to acquire each first attribute name in the first log data to be processed; A first compression name determination module, used for detecting whether there is a compression name corresponding to each first attribute name in the first log data in the mapping relationship between the currently stored attribute name and the compression name; If not, get the current number of compressed names currently in use; If it is detected that the current number is less than the first preset number, a first compressed name corresponding to the first attribute name is generated using a preset character; If it is detected that the current number is greater than or equal to the first preset number and less than the second preset number, a first compressed name corresponding to the first attribute name is generated using two preset characters; The first attribute name and the corresponding first compressed name are stored in the mapping relationship to update the mapping relationship, wherein the first preset number refers to the total number of compressed names that can be generated using one preset character; the second preset number refers to the total number of compressed names that can be generated using two preset characters; A first attribute name replacement module, used for replacing each first attribute name in the first log data with the corresponding first compressed name, to obtain compressed second log data; A log writing module is used to write the second log data into a log file, and write the log file into a full-text search engine, so as to segment and index the log file in the full-text search engine, and perform log search based on log keywords input by the user to obtain search result data, wherein the log keywords include attribute names.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log data compression processing method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the log data compression processing method as described in any one of claims 1 to 8 is implemented.

Images