Log processing method and apparatus, log reading method and apparatus, device and storage medium

Abstract

A log processing method and apparatus, a log reading method and apparatus, a device, and a storage medium. The log processing method comprises: acquiring a log set to be processed, wherein said log set comprises a plurality of logs, and the plurality of logs comprise a same target field; deduplicating values of the plurality of logs in the target field, and writing the obtained values into a first array corresponding to the target field in a log struct; and adding in the log struct a second array corresponding to the target field, on the basis of the value of each log on the target field, determining an index number corresponding to the log in the first array, and sequentially writing corresponding index numbers into the second array on the basis of an arrangement sequence of the logs.

Description

Log processing method, reading method, device, equipment and storage medium Technical Field

[0001] This specification relates to the field of log processing technology, and in particular to a log processing method, reading method, device, equipment, and storage medium. Background Art

[0002] With the development of the internet, the importance of logs has become increasingly prominent across various fields. For example, in cloud computing, logs are used to monitor and audit various cloud services, demonstrating the operational status of systems and networks and enabling diagnosis and location of problems. In the field of big data, logs are a type of massive data. Analysis and mining of this vast amount of data can uncover potential issues and valuable data, providing a reference for management decisions. In the field of network security, various security products and tools generate security logs that can be analyzed by analysis platforms to detect and warn of network attacks and trace their source.

[0003] However, with the development of these fields, the amount of log data has also grown exponentially, and the volume of log data has increased rapidly. While occupying storage, the performance loss problem during log transmission has become increasingly prominent.

[0004] Summary of the Invention

[0005] To overcome the problems existing in the related art, this specification provides a log processing method, a reading method, an apparatus, a device, and a storage medium.

[0006] According to a first aspect of an embodiment of this specification, a log processing method is provided, the method comprising:

[0007] Obtain a set of logs to be processed, where the set of logs to be processed includes multiple logs, and the multiple logs include the same target field;

[0008] Deduplicating the values ​​of the plurality of logs in the target field, and writing the obtained values ​​into a first array corresponding to the target field in the log structure;

[0009] A second array corresponding to the target field is added to the log structure, the index number corresponding to each log in the first array is determined according to the value of each log in the target field, and the corresponding index number is written into the second array in the order of arrangement between the logs.

[0010] Optionally, determining the corresponding index number of each log in the first array based on the value of the target field includes: querying the number of values ​​of the target field in the first array corresponding to the log structure; if the target field has only one value in the first array corresponding to the log structure, then the index number corresponding to each log is set to a predefined special value.

[0011] Optionally, the method further comprises: calling a predefined code, the code being obtained by converting a proto file into a specified programming language;

[0012] A blank log structure is constructed based on the code for writing the to-be-processed log set.

[0013] Optionally, the method further includes: sending the log structure to an analysis platform for log analysis.

[0014] Optionally, the method further includes: compressing the log structure.

[0015] According to a second aspect of an embodiment of this specification, a log reading method is provided, the method comprising:

[0016] Obtain a log structure, wherein the log structure includes a target field, the target field being the same field included in multiple logs in the log structure, the target field corresponding to a first array and a second array in the log structure, the first array storing deduplicated values ​​of the multiple logs in the target field;

[0017] Reading the index number corresponding to each log in the first array corresponding to the target field from the second array corresponding to the target field, wherein the second array stores the index numbers corresponding to the values ​​of the multiple logs in the first array in the order in which the logs are arranged;

[0018] According to the index number, the values ​​of the multiple logs in the target field are read.

[0019] According to a third aspect of an embodiment of this specification, a log processing device is provided, the device comprising:

[0020] A first acquiring unit is configured to acquire a set of logs to be processed, wherein the set of logs to be processed includes multiple logs, and the multiple logs include the same target field;

[0021] A first processing unit is configured to deduplicate the values ​​of the plurality of logs in the target field, and write the obtained values ​​into a first array corresponding to the target field in the log structure;

[0022] An index unit is used to add a second array corresponding to the target field in the log structure, determine the index number corresponding to each log in the first array according to the value of the target field, and write the corresponding index number into the second array in the order of arrangement of the logs.

[0023] According to a fourth aspect of the embodiments of this specification, a log reading device is provided, the device comprising:

[0024] A second acquiring unit is configured to acquire a log structure, wherein the log structure includes a target field, the target field being a common field included in multiple logs in the log structure, the target field corresponding to a first array and a second array in the log structure, the first array storing deduplicated values ​​of the multiple logs in the target field;

[0025] A second processing unit is configured to read, from a second array corresponding to the target field, an index number corresponding to each log entry in the first array corresponding to the target field, wherein the second array stores, in order of arrangement between the log entries, the index numbers corresponding to the values ​​of the multiple log entries in the first array;

[0026] The reading unit is configured to read the values ​​of the target fields of the plurality of logs according to the index number.

[0027] According to a fifth aspect of the embodiments of this specification, an electronic device is provided, including:

[0028] processor;

[0029] a memory for storing processor-executable instructions;

[0030] The processor implements the method described in the embodiment of the first aspect above by running the executable instructions.

[0031] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the embodiment of the first aspect above are implemented.

[0032] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0033] In an embodiment of the present specification, based on the columnar storage idea, the values ​​in at least one identical field of multiple logs in a log set are deduplicated, only the deduplicated values ​​are retained, and an array corresponding to the identical field is added to record the index number of the value of each log in the identical field, thereby converting repeated values ​​into numbers and ensuring that the data of the complete log can still be restored while removing redundant values, thereby reducing the overall volume of the log data. In addition, by adopting the idea of ​​columnar storage to batch process logs, the identical fields of multiple logs are put together, thereby improving the data compression rate and reducing the functional loss during log data transmission.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0036] FIG1 is a flowchart of a log processing method according to an exemplary embodiment of the present specification.

[0037] FIG2 is a schematic diagram of a column-based log storage according to an exemplary embodiment of this specification.

[0038] FIG3 is a schematic diagram of optimized log storage according to an exemplary embodiment of this specification.

[0039] FIG4 is a flowchart of a log reading method according to an exemplary embodiment of this specification.

[0040] FIG5 is a block diagram of a log processing device according to an exemplary embodiment of this specification.

[0041] FIG6 is a block diagram of a log reading device according to an exemplary embodiment of this specification.

[0042] FIG7 is a schematic structural diagram of an electronic device according to an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0044] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0046] With the development of the internet, the importance of logs has become increasingly prominent across various fields. Related technologies often use log formats such as JSON or syslog. However, due to the need to carry field names, enclose strings in double quotes, and separate field names and values ​​with colons or equal signs, these formats often result in large data volumes, occupying a significant amount of log content space. Furthermore, regular expressions are required for log parsing, resulting in significant performance degradation during transmission.

[0047] Some technical solutions use protobuf encoding to optimize log storage. Although field names are not required, redundant values ​​cannot be removed when some fields have redundant values. For example, if each log entry has the same type field with the value "access," 10,000 log entries will have 10,000 "access" field contents. This still results in large log data volumes and high network resource consumption.

[0048] Therefore, this manual provides a log processing method that can reduce the volume of log data and improve log transmission performance.

[0049] Next, the embodiments of this specification are described in detail.

[0050] As shown in FIG1 , FIG1 is a flowchart of a log processing method according to an exemplary embodiment of the present specification, which specifically includes the following steps:

[0051] S101: Obtain a set of logs to be processed, where the set of logs to be processed includes multiple logs, and the multiple logs include the same target field.

[0052] Specifically, by quantitatively reading the currently stored logs, a log set to be processed is obtained, wherein the log set to be processed includes multiple logs, and the same fields exist in the multiple logs, or at least two of the multiple logs have the same fields, and the same fields are used as target fields.

[0053] Among them, by setting a preset number, a predetermined number of logs can be extracted each time to achieve quantitative reading of the currently stored logs; or by setting a preset time, logs generated within a preset time can be extracted each time to achieve quantitative reading of the currently stored logs.

[0054] It's important to note that in the examples of this specification, logs are stored using a column-based storage system. Specifically, logs are stored in columns, with the first column containing the log field name and each subsequent column representing a log entry. This column-based storage allows for quick extraction of fields within the log while reducing disk space usage.

[0055] In some other embodiments, logs may be stored in a conventional manner, and fields in the logs and corresponding log values ​​may be extracted one by one. This specification does not impose any limitation on this.

[0056] In an exemplary embodiment, since the data volume of string type fields is large, while the data volume of numeric type fields themselves is small, the target field only includes string type fields, and other types of strings contained in multiple logs, such as numeric type strings, are not adjusted, thereby reducing the fields that need to be operated during log processing and improving processing efficiency.

[0057] For example, assume that the following log content currently exists in the log collection to be processed:

[0058] Log 1: {"type":"access","timestamp":1700000001,"src_ip":"1.1.1.1","src_port":33331,"dst_ip" :"2.2.2.2", "dst_port": 80, "proto": "http", "url": " / login", "method": "GET", "source": "3.3.3.3"}

[0059] Log 2: {"type":"access","timestamp":1700000002,"src_ip":"1.1.1.1","src_port":33332,"dst_ip": "2.2.2.2", "dst_port": 8080, "proto": "http", "url": " / index", "method": "GET", "source": "3.3.3.3"}

[0060] Log 3 {"type":"access","timestamp":

[0061] 1700000003,"src_ip":"1.1.1.1","src_port":33333,"dst_ip":"2.2.2.2","dst_port":80,"proto":"http","url":" / admin","method":"POST","source":"3.3.3.3"}

[0062] Log 4: {"type":"access","timestamp":

[0063] 1700000004,"src_ip":"1.1.1.1","src_port":33334,"dst_ip":"2.2.2.2","dst_port":8888,"proto":"http","url":" / index","method":"GET","source":"3.3.3.3"}

[0064] The above log contents are displayed in the form of columns to obtain the log contents shown in Figure 2. Figure 2 is a schematic diagram of a columnar log storage shown in this specification according to an exemplary embodiment. Among them, the first column in the dotted box is the field of the log in the above-mentioned log set to be processed, and each subsequent column represents logs 1 to 4 respectively. It can be seen from Figure 2 that all fields in the current log set to be processed are the same fields of the four logs, and since "timestamp", "src_port" and "dst_port" are digital character strings, all fields other than these three fields are used as target fields.

[0065] In an exemplary embodiment, the method further includes: calling predefined code, the code being obtained by converting a proto file into a specified programming language; and constructing a blank log structure based on the code for writing the set of logs to be processed.

[0066] Specifically, users predefine a corresponding proto file based on the set of logs to be processed. Each field is defined as repeated, indicating that each field is an array used to store the values ​​of multiple logs. This proto definition file is converted into code in a specified programming language using protobuf tools. It should be noted that for logs of the same type, if the fields in the logs do not change, the predefined proto file can be reused.

[0067] A blank log structure is constructed by calling the generated code, wherein all fields in the log to be processed are defined in the blank log structure, and arrays corresponding to the fields are used to write log data in the log set to be processed.

[0068] Continuing with the previous example, the log structure is as follows: "syntax="proto2"; packageaccesslog; messageAccessLog{ repeatedstringlogtype=1; repeatedint64timestamp=2; repeatedstringsrc_ip=3; repeatedint32src_port=4; repeatedstringdst_ip=5; repeatedint32dst_port=6; repeatedstringproto=7; repeatedstringurl=8; repeatedstringmethod=9; repeatedstringsource=10;"

[0069] Each field in the above log structure has a unique identification number and is defined as an array to store the values ​​of multiple logs on that field. Fields with the same subscript in each field array belong to the same log. The format of logs stored in the array is as follows, where the log subscript numbers can be predefined according to requirements:

[0070] The values ​​of each field in the xth log are: logtype[x-1], timestamp[x-1], src_ip[x-1], src_port[x-1], dst_ip[x-1], dst_port[x-1], proto[x-1], url[x-1], method[x-1], source[x-1]

[0071] The above values ​​are stored in the arrays corresponding to each field in the log structure.

[0072] S102: Deduplication is performed on the values ​​of the plurality of logs in the target field, and the obtained values ​​are written into a first array corresponding to the target field in the log structure.

[0073] Extract all fields from the log set to be processed. For target fields that are contained in multiple logs, since they correspond to the contents of multiple logs at the same time, set a corresponding array for the target field and fill it with the value corresponding to each log in turn.

[0074] For the array corresponding to each target field, search for duplicate values ​​in the array and deduplicate the duplicate values ​​so that there is only one different value in the array. Write the array corresponding to the target field after deduplication into the array corresponding to the target field in the log structure, and use this array as the first array corresponding to the target field.

[0075] As shown in Figure 2, taking the "method" field as an example, we can see that the value for this target field is repeated in logs 1, 2, and 4: all are "GET." This value is removed, leaving only one. Furthermore, since the value "POST" corresponding to log 3 does not have a duplicate value, no processing is required. After removal of duplicates, the values ​​in the first array corresponding to the "method" field are "GET" and "POST."

[0076] S103: Add a second array corresponding to the target field to the log structure, determine the corresponding index number of each log in the first array according to the value of the target field, and write the corresponding index number into the second array in the order of arrangement of the logs.

[0077] After completing the deduplication operation on the values ​​corresponding to multiple logs in the target field, an index field corresponding to the target field is added to the aforementioned log structure, and the index field corresponds to the second array corresponding to the target field in the log structure.

[0078] The deduplicated values ​​of the target field in the first array are numbered, and the value of each log in the target field is compared with the deduplicated value of the target field in the first array to determine the number of the value of each log in the first array. The number is used as the index number, and the corresponding index number is written into the second array in the order of arrangement of the logs.

[0079] The index field type corresponding to the target field is an integer, which can be a two-byte short or a four-byte int, depending on the maximum number of logs in each batch. If the number is within 65535, a two-byte short is used; if the number is within 255, a char is used; if the number is greater than 65535, an int is used.

[0080] As shown in Figure 3, Figure 3 is a schematic diagram of an optimized log storage according to an exemplary embodiment of this specification. Taking the field "method" as an example, an index field "method_id" corresponding to the "method" field is added to the log structure. This index field corresponds to the second array corresponding to the target field in the log structure. The values ​​"GET" and "POST" in the first array are numbered, assuming they are "1" and "2" respectively.

[0081] Comparing the values ​​in the fields of logs 1 through 4 with the values ​​retained in the first array reveals that the values ​​corresponding to logs 1, 2, and 4 correspond to the "GET" value numbered 1, resulting in an index number of "1" for logs 1, 2, and 4. Furthermore, the value of log 3 corresponds to the "POST" value numbered 2, resulting in an index number of "2" for log 3. The index numbers are then written to the second array in the order of each log entry, followed by "1," "1," "2," and "1." This continues in this manner, resulting in the array corresponding to the target field and the values ​​corresponding to the index field, as shown in Figure 3.

[0082] Write the array corresponding to the target field in the log set to be input into the log structure in the above form. Similarly, for other fields other than the target field, write the corresponding log values ​​into the log structure, thereby completing the writing of the log set to be processed in the log structure.

[0083] Continuing with the above example, the log structure is: “syntax="proto2"; packageaccesslog; messageAccessLog{ repeatedstringlogtype=1; repeatedint32logtype_id=2; repeatedint64timestamp=3; repeatedstringsrc_ip=4; repeatedint32src_ip_id=5; repeatedint32src_port=6; repeatedstringdst_ip=7; repeatedint32dst_ip_id=8; repeatedint32dst_port=9; repeatedstringproto=10; repeatedint32proto_id=11; repeatedstringurl=12; repeatedint32url_id=13; repeatedstringmethod=14; repeatedint32method_id=15; repeatedstringsource=16; repeatedint32source_id=17;”

[0084] The first array and the second array corresponding to each target field are respectively written with the deduplicated value and index number.

[0085] In an exemplary embodiment, the number of values ​​of the target field in the first array corresponding to the log structure is queried. If the target field has only one value in the first array corresponding to the log structure, the index number corresponding to each log is set to a predefined special value.

[0086] Specifically, after completing the deduplication of the target field, the number of remaining values ​​of the target field in the first array corresponding to the log structure is queried. If the target field has only one value in the first array corresponding to the log structure, it means that the values ​​of the target field in the multiple logs containing the target field are all the same. The index number corresponding to each log is set to a predefined special value to indicate that the values ​​of the logs corresponding to the current field are all the same, thereby avoiding indexing each value in the target field and then obtaining the value, thereby improving processing efficiency.

[0087] As shown in Figure 3, taking the field "type" as an example, there is only one value in the first array corresponding to the current field after deduplication, which means that the values ​​of multiple logs on this field are all the same. In this case, a predefined special value can be used when defining the index number, such as "0". Then, when reading the current log, when the index number corresponding to the value in the field is queried as "0", it is considered that the values ​​in all logs in this field are the same value, and there is no need to query the value of each log one by one.

[0088] By adopting the idea of ​​columnar storage to process logs in batches, the same fields of multiple logs are put together, and duplicate values ​​are removed and indexes are added. This reduces the volume of stored log data, improves the data compression rate, and reduces the functional loss during log data transmission.

[0089] In an exemplary embodiment, after a complete log structure is obtained, the log structure is sent to an analysis platform via a network for log analysis.

[0090] In an exemplary embodiment, the log structure is compressed before being sent. Specifically, because the values ​​of the same field in multiple logs are located in the same array and belong to continuous memory, the compression rate can be improved, further reducing the data volume corresponding to the log.

[0091] In an embodiment of the present specification, based on the columnar storage idea, the values ​​in at least one identical field of multiple logs in a log set are deduplicated, only the deduplicated values ​​are retained, and an array corresponding to the identical field is added to record the index number of the value of each log in the identical field, thereby converting repeated values ​​into numbers and ensuring that the data of the complete log can still be restored while removing redundant values, thereby reducing the overall volume of the log data. In addition, by adopting the idea of ​​columnar storage to batch process logs, the identical fields of multiple logs are put together, thereby improving the data compression rate and reducing the functional loss during log data transmission.

[0092] As shown in FIG4 , FIG4 is a flowchart of a log reading method according to an exemplary embodiment of this specification, which specifically includes the following steps:

[0093] S401, obtain a log structure, wherein the log structure includes a target field, the target field is the same field included in multiple logs in the log structure, and the target field corresponds to a first array and a second array in the log structure, and the first array stores the deduplicated values ​​of the multiple logs in the target field.

[0094] S402: Reading the index number corresponding to each log entry in the first array corresponding to the target field from the second array corresponding to the target field, wherein the second array stores the index numbers corresponding to the values ​​of the plurality of log entries in the first array in the order in which the log entries are arranged;

[0095] S403: Read the values ​​of the target fields of the multiple logs according to the index number.

[0096] Specifically, multiple logs in the log structure are read sequentially. When reading the first log, the value in the first array corresponding to the first field is first read. Then, the referenced number in the second array under the index field corresponding to the first field is read. Assuming that the index number of the first field in the first log is "1", the value corresponding to the index number "1" in the first array is read as the value of the first field in the first log. This process is repeated until every field in each log is deduced.

[0097] 1 , the fields in the obtained log structure and the corresponding array data are shown in FIG3 . Assuming that there is only one value left in the first array of the target field, the special value of the index number is set to “0”.

[0098] It should be noted that in the embodiments of this specification, the first array corresponding to each field stores the deduplicated values, and the second array stores the index code. The index number of the x-th log on the first field is defined as logtype_id[x-1], and the same applies to other fields.

[0099] The reading process of the first log includes:

[0100] Read the value of the first field "logtype": First, read the index code corresponding to the first log in the second array corresponding to the first field, that is, logtype_id[0]. If the corresponding index code is "0", it proves that the values ​​of all logs corresponding to the first field are the same. There is only one value in the first array. Directly read the value "access" of logtype[0] in the first array.

[0101] Read the value of the second field "timestamp": Since the second field is not a string type field, directly read the value of timestamp[0].

[0102] Similarly, read the value in the eighth field "url": first read the index code corresponding to url_id[0]. If the corresponding index code is "1", then read the value corresponding to url[url_id[0]-1], that is, url[1-1]. The value of url[0] is " / login".

[0103] According to the above algorithm, all values ​​of multiple fields in the first log are obtained.

[0104] The process of reading the xth log includes:

[0105] Read the value in the y-th field: First read the index code corresponding to "y index field name [x-1]". If the corresponding index code is "0", directly read the value of "y index field name [0]" in the first array. If the corresponding index code is greater than 0, read the value corresponding to "y field name [y index field name [x-1] - 1].

[0106] In some other embodiments, those skilled in the art can set the numbers of the first array and the second array of the target field as needed, and obtain the final log value through other calculation methods, but the basic idea of ​​column-based storage remains unchanged and this specification does not limit this.

[0107] For example, if the special value setting is canceled, the index number of the xth log in the first field is defined as logtype_id[x], and the first value in the first array is logtype[1]. Then, the method for reading the value of the first field of the first log is to read the value of logtype[logtype_id[x]].

[0108] In an exemplary embodiment, after the log receiving end receives the log content, if it is compressed when sent, the log content is first decompressed to obtain a log structure.

[0109] Through this embodiment, the log structure in the embodiment corresponding to FIG1 can be quickly read to obtain complete log set data.

[0110] Corresponding to the aforementioned method embodiments, this specification also provides device embodiments.

[0111] As shown in FIG5 , FIG5 is a block diagram of a log processing device according to an exemplary embodiment of this specification, the device including:

[0112] A first acquiring unit 501 is configured to acquire a set of logs to be processed, wherein the set of logs to be processed includes multiple logs, and the multiple logs include the same target field.

[0113] A first processing unit 502 is configured to deduplicate the values ​​of the target field in the plurality of logs, and write the obtained values ​​into a first array corresponding to the target field in the log structure;

[0114] The index unit 503 is used to add a second array corresponding to the target field in the log structure, determine the corresponding index number in the first array according to the value of each log in the target field, and write the corresponding index number into the second array in the order of arrangement between the logs.

[0115] In an exemplary embodiment, determining the corresponding index number of each log in the first array based on the value of the target field includes: querying the number of values ​​of the target field in the first array corresponding to the log structure; if the target field has only one value in the first array corresponding to the log structure, then the index number corresponding to each log is set to a predefined special value.

[0116] In an exemplary embodiment, the device further includes: a calling unit 504, configured to call a predefined code, the code being obtained by converting a proto file into a specified programming language; and constructing a blank log structure based on the code for writing the set of logs to be processed.

[0117] In an exemplary embodiment, the apparatus further includes: a sending unit 506, configured to send the log structure to an analysis platform for log analysis.

[0118] In an exemplary embodiment, the apparatus further includes: a compression unit 505, configured to compress the log structure.

[0119] As shown in FIG6 , FIG6 is a block diagram of a log reading device according to an exemplary embodiment of this specification, the device including:

[0120] A second acquiring unit 601 is configured to acquire a log structure, wherein the log structure includes a target field, which is a common field included in multiple logs in the log structure. The target field corresponds to a first array and a second array in the log structure, wherein the first array stores deduplicated values ​​of the multiple logs in the target field.

[0121] The second processing unit 602 is configured to read the index number corresponding to each log entry in the first array corresponding to the target field from the second array corresponding to the target field, wherein the second array stores the index numbers corresponding to the values ​​of the multiple log entries in the first array in the order in which the log entries are arranged;

[0122] The reading unit 603 is configured to read values ​​of the target field of the plurality of logs according to the index number.

[0123] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0124] FIG7 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of this specification. Referring to FIG7 , at the hardware level, the electronic device includes a processor 701, an internal bus 702, a network interface 703, a memory 704, and a non-volatile memory 705, and may also include hardware required for other services. The processor 701 reads the corresponding computer program from the non-volatile memory 705 into the memory 704 and then runs it. Of course, in addition to software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0125] The devices or modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0126] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, a log processing method and a complaint method as shown in any of the aforementioned embodiments are implemented.

[0127] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0128] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0129] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0130] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0131] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A log processing method, characterized in that: include: Obtain a set of logs to be processed, where the set of logs to be processed includes multiple logs, and the multiple logs include the same target field; Deduplicating the values of the plurality of logs in the target field, and writing the obtained values into a first array corresponding to the target field in the log structure; A second array corresponding to the target field is added to the log structure, the index number corresponding to each log in the first array is determined according to the value of each log in the target field, and the corresponding index number is written into the second array in the order of arrangement between the logs.

2. The method according to claim 1, characterized in that Determining the index number corresponding to each log in the first array according to the value of each log in the target field includes: The number of values of the target field in the first array corresponding to the log structure is queried. If the target field has only one value in the first array corresponding to the log structure, the index number corresponding to each log is set to a predefined special value.

3. The method according to claim 1, characterized in that The method further comprises: Call predefined code, which is obtained by converting the proto file into a specified programming language; A blank log structure is constructed based on the code for writing the to-be-processed log set.

4. The method according to claim 1, wherein The method further comprises: The log structure is sent to the analysis platform for log analysis.

5. The method according to claim 1, wherein The method further comprises: The log structure is compressed.

6. A log reading method, characterized in that: include: Obtain a log structure, wherein the log structure includes a target field, the target field being the same field included in multiple logs in the log structure, the target field corresponding to a first array and a second array in the log structure, the first array storing deduplicated values of the multiple logs in the target field; Reading the index number corresponding to each log in the first array corresponding to the target field from the second array corresponding to the target field, wherein the second array stores the index numbers corresponding to the values of the multiple logs in the first array in the order in which the logs are arranged; According to the index number, the values of the target field of the multiple logs are read.

7. A log processing device, characterized in that: include: A first acquiring unit is configured to acquire a set of logs to be processed, wherein the set of logs to be processed includes multiple logs, and the multiple logs include the same target field; A first processing unit is configured to deduplicate the values of the plurality of logs in the target field, and write the obtained values into a first array corresponding to the target field in the log structure; An index unit is used to add a second array corresponding to the target field in the log structure, determine the index number corresponding to each log in the first array according to the value of the target field, and write the corresponding index number into the second array in the order of arrangement of the logs.

8. A log reading device, characterized in that: include: The second acquisition unit is used to acquire a log structure, wherein the log structure includes a target field, and the target field is the same field included in multiple logs in the log structure, and the target field corresponds to a first array and a second array, wherein the first array stores deduplicated values of the plurality of logs in the target field; A second processing unit is configured to read, from a second array corresponding to the target field, an index number corresponding to each log entry in the first array corresponding to the target field, wherein the second array stores, in order of arrangement between the log entries, the index numbers corresponding to the values of the multiple log entries in the first array; The reading unit is configured to read the values of the target fields of the plurality of logs according to the index number.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images