Log processing method and apparatus, electronic device, storage medium, and program product

Abstract

The present application relates to the technical field of data processing, and provides a log processing method and apparatus, an electronic device, a storage medium, and a program product. When a log to be written is a long log having a length that exceeds a retrieval limit length, the log needs to be truncated and then stored; among log segments obtained by truncation, every two adjacent log segments have repeated bytes of a second preset length; on the basis of the overlapping portion of the adjacent log segments, at least one log segment contains complete key information, so that the log segments obtained by truncation can support retrieval by means of a keyword. When a user reads the long log, a log storage system can retrieve corresponding log segments on the basis of the keyword and acquire the related log segments, splice the plurality of log segments to restore the long log, and return the long log to the user.

Description

Log processing methods, devices, electronic equipment, storage media, and software products

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 2024114568911, filed on October 17, 2024, entitled "Log Processing Method, Apparatus, Electronic Device, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of data processing technology, and in particular to a log processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0004] Logging is a very important practice in software development. Logging is like taking notes on the software's operation process, recording various events that occur during the software's operation, so that technical personnel can better understand the software's behavior, troubleshoot problems, and optimize performance.

[0005] Different programming languages ​​have different logging frameworks, such as Log4j and Logback for Java, and logging for Python. Logs can be processed in intermediate layers or stored directly in a storage engine that supports full-text search. Taking Elasticsearch as an example, in order to achieve accurate matching when searching logs, Elasticsearch typically uses the keyword type to store logs. However, the keyword type has a length limit. Even if the length limit is greatly increased, it is difficult to predict how long the user will record the logs. Logs longer than the limit can only be stored and cannot be filtered out by retrieving other fields of the structured logs. More importantly, logs longer than the keyword limit cannot be indexed and cannot be retrieved by keyword search. Summary of the Invention

[0006] This application provides a log processing method, apparatus, electronic device, storage medium, and program product.

[0007] A first aspect of this application provides a log processing method, wherein the method includes:

[0008] Receive logs to be written;

[0009] Determine the length of the log to be written;

[0010] If the length of the log to be written is greater than the first preset length, the log to be written is truncated to obtain at least two segmented logs, and each pair of adjacent segmented logs contains repeated characters of a second preset length; the second preset length is determined based on the length of the keywords that support retrieval.

[0011] Add a truncation identifier and a segment number to the at least two segmented log segments;

[0012] Store at least two segmented log segments carrying the truncation identifier and segment number.

[0013] Optionally, the log to be written is truncated to obtain at least two segmented log segments, including:

[0014] The starting position of the log to be written is taken as the starting position of the first segment log, and the ending position of the first segment log is obtained by excluding a third preset length.

[0015] Starting from the end position of the first segment log and moving towards the start position, the start position of the second segment log is obtained by pausing at a second preset length. Starting from the start position of the second segment log, the end position of the second segment log is obtained by pausing at a third preset length, and so on, until the end position of the log to be written is reached. The end position of the log to be written is then taken as the end position of the last segment log.

[0016] Based on the start and end positions of each segmented log, at least two segmented logs are obtained.

[0017] Optionally, both the second preset length and the third preset length are smaller than the first preset length.

[0018] Optionally, the method further includes:

[0019] Receive a log search request, the log search request carrying search keywords;

[0020] The first log associated with the search keyword is obtained by searching based on the search keyword;

[0021] If the first log carries a truncation identifier, obtain the second log related to the first log. The first log and the second log are multiple segmented logs obtained by truncating the same log.

[0022] Based on the segment sequence numbers of the first log and the second log, the first log and the second log are concatenated to obtain the complete target log;

[0023] Send the complete target log.

[0024] Optionally, the first log and the second log are concatenated according to the segment sequence numbers of the first log and the second log to obtain a complete target log, including:

[0025] The order of each segment log is obtained by sorting the segment numbers of the first and second logs.

[0026] The second preset length is removed from the end of each segment log from the first to the second to last segment log; multiple trimmed segment logs are obtained.

[0027] The multiple cropped log segments and the last log segment are concatenated according to the segment number to obtain the complete target log.

[0028] Optionally, after adding truncation identifiers and fragment numbers to the at least two segmented log segments, the method further includes: adding digital digests to the at least two segmented log segments;

[0029] After obtaining the complete target log, the method further includes:

[0030] The complete target log is verified based on the digital digest;

[0031] Send the complete target log, including:

[0032] Send the complete target log that has passed verification.

[0033] Optionally, before truncating the log to be written, the method further includes:

[0034] Receive the length configuration information of the keywords that support retrieval;

[0035] The second preset length is determined based on the length of the keywords that support retrieval;

[0036] After adding truncation identifiers and fragment numbers to the at least two segmented log segments, the process also includes:

[0037] Add a second preset length configuration information to the at least two segmented logs.

[0038] A second aspect of this application provides a log processing method, the method comprising:

[0039] Send the length configuration information of the keywords that support retrieval, the length configuration information being used to determine the second preset length;

[0040] Send a log storage request to the log storage system, the log storage request carrying the logs to be written;

[0041] The log storage system is used to truncate the log to be written when the log to be written is greater than a first preset length, to obtain at least two segmented logs, add a truncation identifier and a segment number to the at least two segmented logs, and store the at least two segmented logs carrying the truncation identifier and the segment number, wherein each pair of adjacent segmented logs contains repeated characters of the second preset length.

[0042] A third aspect of this application provides a log processing apparatus, the apparatus comprising:

[0043] The first receiving module is used to receive logs to be written.

[0044] The judgment module is used to determine the length of the log to be written;

[0045] The truncation module is used to truncate the log to be written when the length of the log to be written is greater than a first preset length, to obtain at least two segmented logs, wherein each pair of adjacent segmented logs contains repeated characters of a second preset length; the second preset length is determined based on the length of the keywords that support retrieval;

[0046] The first adding module is used to add truncation identifiers and fragment numbers to the at least two segments of log;

[0047] A storage module is used to store at least two segmented logs carrying the truncation identifier and segment number.

[0048] Optionally, the truncation module is specifically used for:

[0049] The starting position of the log to be written is taken as the starting position of the first segment log, and the ending position of the first segment log is obtained by excluding a third preset length.

[0050] Starting from the end position of the first segment log and moving towards the start position, the start position of the second segment log is obtained by pausing at a second preset length. Starting from the start position of the second segment log, the end position of the second segment log is obtained by pausing at a third preset length, and so on, until the end position of the log to be written is reached. The end position of the log to be written is then taken as the end position of the last segment log.

[0051] Based on the start and end positions of each segmented log, at least two segmented logs are obtained.

[0052] Optionally, the device further includes:

[0053] The second receiving module is used to receive log search requests, wherein the log search requests carry search keywords;

[0054] The search module is used to search for and obtain the first log associated with the search keywords;

[0055] The acquisition module is used to acquire a second log related to the first log when the first log carries a truncation identifier. The first log and the second log are multiple segmented logs obtained by truncating the same log.

[0056] The splicing module is used to splice the first log and the second log according to the segment sequence number of the first log and the second log to obtain the complete target log;

[0057] The sending module is used to send the complete target log.

[0058] Optionally, the splicing module is specifically used for:

[0059] The order of each segment log is obtained by sorting the segment numbers of the first and second logs.

[0060] The second preset length is removed from the end of each segment log from the first to the second to last segment log; multiple trimmed segment logs are obtained.

[0061] The multiple cropped log segments and the last log segment are concatenated according to the segment number to obtain the complete target log.

[0062] Optionally, the device further includes:

[0063] The second adding module is used to add digital digests to the at least two segmented log segments;

[0064] The verification module is used to verify the complete target log based on the digital digest;

[0065] The verification module is specifically used for:

[0066] Send the complete target log that has passed verification.

[0067] Optionally, the device further includes:

[0068] The third receiving module is used to receive the length configuration information of the keywords that support retrieval;

[0069] The determining module is used to determine a second preset length based on the length of the keywords that support retrieval;

[0070] The third adding module is used to add a second preset length configuration information to the at least two segmented logs.

[0071] A fourth aspect of this application provides a log processing apparatus, the apparatus comprising:

[0072] The first sending module is used to send length configuration information of keywords that support retrieval, wherein the length configuration information is used to determine a second preset length;

[0073] The second sending module is used to send a log storage request to the log storage system, wherein the log storage request carries the log to be written;

[0074] The log storage system is used to truncate the log to be written when the log to be written is greater than a first preset length, to obtain at least two segmented logs, add a truncation identifier and a segment number to the at least two segmented logs, and store the at least two segmented logs carrying the truncation identifier and the segment number, wherein each pair of adjacent segmented logs contains repeated characters of the second preset length.

[0075] The fifth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the log processing method as described in the first or second aspect of this application.

[0076] A sixth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log processing method as described in the first or second aspect of this application.

[0077] A seventh aspect of this application provides a computer program product, including a computer program / instructions, which are implemented by a processor as the steps in the log processing method described in the first or second aspect of this application.

[0078] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0079] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 is a flowchart of the log processing method provided in an embodiment of this application;

[0081] Figure 2 is a schematic diagram of segmented logs in the log processing method provided in the embodiment of this application;

[0082] Figure 3 is a schematic diagram of the overall flow of the log processing method provided in the embodiment of this application;

[0083] Figure 4 is a schematic diagram of the log search process of the log processing method provided in the embodiments of this application;

[0084] Figure 5 is a flowchart of the log processing method provided in an embodiment of this application;

[0085] Figure 6 is a schematic diagram of the log processing device provided in an embodiment of this application;

[0086] Figure 7 is a schematic diagram of the log processing device provided in an embodiment of this application. Detailed Implementation

[0087] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0088] One solution to the problem of how to store long logs is to truncate them. However, if important information is truncated, it cannot be found in the search. When users search for logs, they cannot achieve accurate matching. For users, it is as if the log was not stored at all, thus losing its purpose.

[0089] One solution used in related technologies is to store long logs using the text type. However, this approach can lead to the log storage system being unable to accurately match the logs. When users search based on keywords, they will see many irrelevant results, which affects the search experience.

[0090] Based on this, this application proposes to truncate long logs into segments shorter than the keyword limit for storage. Furthermore, based on the length of the searchable keyword, overlapping portions are retained among the multiple truncated log segments. This solves the problem of unsearchable long logs when stored using keywords, and is superior to methods that directly truncate or store the log text using a text type.

[0091] Specifically, this application proposes a log processing method, as shown in Figure 1, which illustrates a flowchart of the steps of the log processing method proposed in this application. The method includes the following steps:

[0092] S101, Receive logs to be written.

[0093] S102, determine the length of the log to be written.

[0094] S103, if the length of the log to be written is greater than the first preset length, the log to be written is truncated to obtain at least two segmented logs, and each pair of adjacent segmented logs contains repeated characters of a second preset length.

[0095] S104, add a truncation identifier and a segment number to the at least two segmented log segments.

[0096] S105, store at least two segmented logs carrying the truncation identifier and segment number.

[0097] In this embodiment, the method can be applied to a log storage system, such as Elasticsearch. Elasticsearch is an open-source distributed search and analytics engine. It is based on the Apache Lucene library and written in Java. Elasticsearch can efficiently store, search, and analyze massive amounts of data, and is particularly adept at handling unstructured data, such as text, logs, and geospatial data.

[0098] In this embodiment of the application, the user can initiate a log storage request through the write gateway of the log storage system and carry the log to be written in the request.

[0099] In this embodiment of the application, the log to be written may include a text file containing a series of events recorded by the software during its operation. These events may include: error messages: error messages generated when the software encounters problems during operation; warning messages: potential problems or abnormal situations; debugging information: detailed records used by developers to track the program execution flow; user operations: operations performed by the user in the software, such as logging in, clicking buttons, etc.; system information: the system status during software operation, such as memory usage, CPU utilization, etc.

[0100] In this embodiment of the application, the length of the characters to be written to the log can be determined. If the length of the log to be written is greater than the first preset length, it is determined to be a long log and needs to be truncated before storage.

[0101] In this embodiment of the application, the first preset length is the retrieval limit length. Long logs that exceed the retrieval limit length will not be indexed after storage and will not be able to be filtered out by other fields of the structured log. For example, the length can be limited to the keyword. Specifically, the first preset length can be 1024 characters.

[0102] Therefore, this application proposes to use the truncation step in step S103 to retain the overlapping part of every two adjacent log segments, so as to avoid important information being truncated and thus unable to be retrieved later.

[0103] Specifically, in this embodiment of the application, step S103 may include the following sub-steps:

[0104] S1031, take the starting position of the log to be written as the starting position of the first segment log, and obtain the ending position of the first segment log by an interval of a third preset length.

[0105] S1032, starting from the end position of the first segment log and moving towards the start position, the start position of the second segment log is obtained at a second preset length interval. Starting from the start position of the second segment log, the end position of the second segment log is obtained at a third preset length interval, and so on, until the end position of the log to be written is reached. The end position of the log to be written is then taken as the end position of the last segment log.

[0106] S1033 obtains at least two segmented logs according to the start and end positions of each segmented log.

[0107] For ease of understanding, this application provides a segmented log diagram, as shown in Figure 2. In Figure 2, 0 represents the starting position of the log to be written, M represents the starting position of the second segmented log, N represents the ending position of the first segmented log, and ~ indicates that the log is not yet finished. Subsequent segmentation follows the same method as the second segment. NM is determined based on the length of the supported search keywords. In this application embodiment, both M and N can be configured by the user.

[0108] In this embodiment of the application, after determining the start and end positions of each segment log, the content to be written to the corresponding position of each segment log can be copied based on the start and end positions of each segment log to obtain each segment log, and then each segment log is stored in the log storage system.

[0109] In this embodiment, during the truncation process, the log can first be truncated to obtain the first segment log according to a third preset length. Then, a second preset length of segment log is copied from the end position of the first segment log towards the start position. This segment log is added to the start position of the remaining log after truncating the first segment log in the order of the original log content to be written. Then, the log is truncated from the start position of the remaining position to obtain the second segment log according to the third preset length, and so on, until the length of the remaining log does not exceed the third preset length. This also allows for the truncation of at least two segment logs, with each adjacent pair of segment logs containing a second preset length of repeated characters.

[0110] In this embodiment of the application, no specific restrictions are placed on the truncation strategy during the truncation process. The restriction is that in at least two segments of log obtained after truncation, a second preset length of repeated characters is stored between each two adjacent segments of log.

[0111] In this embodiment, the third preset length (such as the N value in the example above) is less than the first preset length, so as to truncate the long log into segmented logs that do not exceed the retrieval limit length (the first preset length), so that the stored segmented logs can be retrieved. For example, if the first preset length is 1024 characters, then the third preset length can be 1000 characters.

[0112] In this embodiment of the application, the third preset length can be specified by the user according to their actual needs when initiating a log storage request.

[0113] In this embodiment, the second preset length can be the length of the keywords that support retrieval, generally set to one-tenth of the first preset length. Alternatively, it can be specified by the user when initiating a log storage request based on their actual needs. For example, if the user subsequently needs to retrieve the log using an 11-character user's mobile phone number, the second preset length can be set to 11 characters. In this case, at least one of the multiple segmented logs obtained after truncating the long log will contain the complete user's mobile phone number, enabling accurate log retrieval.

[0114] In this embodiment of the application, in order to enable smooth searching and splicing of segmented logs in the future, it is proposed to add truncation identifiers and segment numbers to the segmented logs.

[0115] Specifically, in this embodiment of the application, a truncation identifier can be added to a specific field for segmented logs. For example, a UUID can be used to fill the field, and then all segmented logs contained in the long log can be found based on the truncation identifier.

[0116] In this embodiment of the application, the truncation identifier is unique for multiple segmented logs obtained by truncating a long log segment.

[0117] In this embodiment of the application, the truncation identifier can indicate that the log fragment stored in the log storage system is a segmented log, and can also be used to find all segmented logs of the same long log.

[0118] In this embodiment of the application, a segment number can also be added to the segmented log in a specific field so that the segmented logs can be spliced ​​together in order according to the segment number of each segmented log to obtain a complete long log.

[0119] It should be noted that, in this embodiment of the application, although there are repeated characters between every two adjacent segments in the segmented logs stored in the log storage system, the length limit for retrieval can be increased, for example, to 1024 characters. In this case, the number of long logs exceeding the retrieval limit is small, and for the overall log storage system, it will not increase the waste of too many storage resources.

[0120] In this embodiment of the application, after a user writes logs through the write gateway of the log storage system and completes the log storage, the user can also read the stored logs through the read gateway of the log storage system.

[0121] Specifically, in this embodiment of the application, the method further includes the following steps:

[0122] S106, Receive a log search request, the log search request carrying search keywords.

[0123] S107, retrieve the first log associated with the search keyword based on the search keyword.

[0124] S108, if the first log carries a truncation identifier, obtain the second log related to the first log, wherein the first log and the second log are multiple segmented logs obtained by truncating the same log.

[0125] S109, according to the segment sequence number of the first log and the second log, the first log and the second log are concatenated to obtain the complete target log.

[0126] S110, send the complete target log.

[0127] Specifically, in this embodiment of the application, a user can initiate a log search request through the read gateway of the log storage system. The search request carries a search keyword, and the read gateway can search the log storage system for all logs associated with the search keyword. These logs may be complete logs or segmented logs that have been truncated and stored.

[0128] In this embodiment of the application, when a searched log carries a truncation identifier, it indicates that the log is a segmented log of a long log. Other segmented logs related to this log can then be found based on the truncation identifier. These segmented logs can be concatenated into a complete log segment.

[0129] In this embodiment of the application, after the search and concatenation are completed, all complete target logs associated with the search keyword can be returned to the user. Thus, the user can accurately find and read long logs by searching for the keyword.

[0130] For ease of understanding, Figure 3 illustrates the overall writing and reading process of the log processing method provided in this embodiment. Figure 3 shows a schematic diagram of the overall flow of the log processing method provided in this embodiment. As shown in Figure 3, in this embodiment, the user can write long logs to the log storage system (Elasticsearch is used as an example in the figure) through a write gateway. The write gateway can truncate the long log to obtain at least two segmented logs. Each pair of adjacent segmented logs contains repeated characters of a second preset length, thus writing the long log into the log storage system in segments. During this process, the user is unaware of this, and for the user, the storage of the long log is successfully completed.

[0131] When logs need to be searched and read, users can initiate a search request to the log storage system through the read gateway based on search keywords. The read gateway then searches the log storage system for all logs related to the search keywords. If there are logs carrying truncation identifiers, all segmented logs related to those logs can be obtained based on the truncation identifier (e.g., UUID). These segmented logs are then concatenated according to concatenation rules to obtain the complete target log. Finally, all complete target logs associated with the search keywords are returned to the user.

[0132] Therefore, in this embodiment of the application, when a user reads a long log, the log storage system can retrieve the corresponding segmented log based on keywords and obtain other segmented logs related to the segmented log, concatenate multiple segmented logs to restore the long log, and return the long log to the user, thereby enabling the user to read the long log visually without being aware of it.

[0133] In this embodiment of the application, the process of splicing the log segments in step S109 above may specifically include the following sub-steps:

[0134] S1091, sort the log segments according to the segment numbers of the first log and the second log to obtain the order of each segment log.

[0135] S1092, trim the end of each segment log from the first to the second to last segment log to remove the second preset length; to obtain multiple trimmed segment logs.

[0136] S1093, the multiple cropped segmented logs and the last segmented log are concatenated according to the segment number to obtain the complete target log.

[0137] Specifically, in this embodiment of the application, the end of all logs except the last segment log can be trimmed to a second preset length to avoid repetition.

[0138] In this embodiment, the write gateway can also add a digital digest to the segmented logs to facilitate subsequent content verification of the concatenated logs. Specifically, in step S104, a digital digest can be added to at least two truncated segmented logs. In this embodiment, the digital digest is generated based on the log content of the original log to be written.

[0139] In this embodiment of the application, after the read gateway obtains all segmented logs of the same long log, it can also obtain the digital digest of the long log from a preset field. Then, after step S1093, the method further includes: verifying the complete target log according to the digital digest.

[0140] In this embodiment of the application, in step S110, the complete target log that has passed verification is sent. For the concatenated log that has failed verification, a prompt tag indicating that the integrity verification failed can be added, and the concatenated log carrying the prompt tag can be returned.

[0141] In this embodiment of the application, the user can also configure the second preset length. Specifically, referring to Figure 2, the user can configure the value of NM, and the log storage system can determine the values ​​of M and N, thereby determining the start and end positions of each segmented log and completing the truncation of long logs.

[0142] Specifically, in this embodiment of the application, before step S103, the method may further include:

[0143] S1 receives the length configuration information of the keywords that support retrieval.

[0144] S2, determine the second preset length based on the length of the keywords that support retrieval.

[0145] In this scenario, after adding a truncation identifier and fragment number to the segmented log in step S104, it is also necessary to add a second preset length configuration information to the segmented log. Thus, the read gateway can complete the segmented log concatenation based on the found second preset length configuration information.

[0146] For ease of understanding, the reading process of the log processing method provided in this application embodiment is described below with reference to Figure 4. Figure 4 shows a schematic diagram of the log search process of the log processing method provided in this application embodiment. As shown in Figure 4, in this application embodiment, the user can initiate a log search request to the read gateway of the log storage system by searching for the keyword K. The read gateway can then search for all fragments containing K from the log storage system (Elasticsearch is used as an example in the figure). The log storage system returns all the fragments containing K to the read gateway. Among these fragments, if there are log fragments carrying truncation identifiers, the read gateway further searches for log fragments containing the truncation identifier (UUID) from all log fragments stored in the log storage system.

[0147] Specifically, in this embodiment of the application, the segment containing K may contain multiple log segments carrying truncation identifiers. Since the truncation identifier corresponding to each long log segment is unique after being truncated, in this embodiment of the application, the read gateway can concurrently search for all log segments from the log storage system based on each unique truncation identifier.

[0148] For a log segment containing a truncation identifier named K, after searching for all segments containing that identifier, they can be sorted based on their segment numbers. For example, the segment numbers can be S1, S2, S3...SN. In this embodiment, for log segments with subsequent numbers (i.e., all log segments except the last one), the ends of these log segments can be trimmed according to a second preset length (NM) determined by pre-parameterized configuration, and the trimmed log segments S1', S2', S3'...SN can be concatenated. Then, the concatenated log can be verified based on the digital digest, and all complete logs containing K (including the concatenated complete target log and the untrunculated original log) can be returned to the user.

[0149] Compared to ordinary truncation, this embodiment adds a second preset length that can be parameterized and configured during the log truncation process. Based on the overlap of the second preset length between adjacent log segments, the truncated log segments can still be retrieved. In a normal truncation scheme, if the truncated part is exactly a phone number or user ID, a user searching using that phone number or user ID will not be able to find the truncated log. However, in this embodiment, because the second preset length overlaps between multiple log segments, even if the phone number or user ID is truncated in the previous log segment, it will still be complete in the next log segment, effectively supporting retrieval.

[0150] Furthermore, in this embodiment, the second preset length is configurable by parameters, and this configuration information can be added to a specified field of the truncated log segments. For example, it can be added to a fixed number of bits after the log segment, such as using a bitmap. Thus, users can parameterize the second preset length during the long log truncation process according to their actual needs. Even when the second preset length changes continuously, the read gateway can still accurately complete the search and splicing of the log segments. Therefore, this embodiment provides an end-to-end, user-unobtrusive solution from writing to visual reading to address the problem that long logs can only be stored but not searched.

[0151] Based on the same inventive concept, this application also provides a log processing method, as shown in Figure 5, which illustrates the flowchart of the log processing method provided in this application. Specifically, the method includes the following steps:

[0152] S501, Send the length configuration information of the keywords that support retrieval, the length configuration information being used to determine the second preset length.

[0153] S502, a log storage request is sent to the log storage system, the log storage request carrying logs to be written.

[0154] The log storage system is used to truncate the log to be written when the log to be written is greater than a first preset length, to obtain at least two segmented logs, add a truncation identifier and a segment number to the at least two segmented logs, and store the at least two segmented logs carrying the truncation identifier and the segment number, wherein each pair of adjacent segmented logs contains repeated characters of the second preset length.

[0155] In this embodiment, the log processing method is used on a client. A user can send length configuration information for keywords that support retrieval to the log storage system via the client. If the length of the log uploaded by the user through the client exceeds the retrieval limit, the log storage system can determine a second preset length based on the length configuration information of the retrieved keywords. Then, based on the second preset length, the long log is truncated into segments shorter than the retrieval limit, and each pair of adjacent segments contains repeated characters of the second preset length. Therefore, in this embodiment, long logs can be segmented and stored for retrieval. Because the second preset length overlaps between multiple segments, even if key information is truncated in one segment, the key information remains complete in the next segment, effectively supporting retrieval.

[0156] In this embodiment, the second preset length can be configured by the user through the client by sending the length configuration information of the keywords that support retrieval. Thus, the user can parameterize the second preset length according to actual needs. Of course, in this embodiment, the second preset length can also be pre-configured by technical personnel, so that the user does not need to configure the second preset length every time the log is updated.

[0157] In this embodiment, the specific log truncation steps can be referred to the above embodiments, and will not be repeated here.

[0158] In this embodiment of the application, the method may further include the following steps:

[0159] S503, a log search request is sent to the log storage system, the log search request carrying search keywords.

[0160] S504, receive log search results.

[0161] The log storage system is used to search for a first log associated with the search keyword; if the first log carries a truncation identifier, it obtains a second log associated with the first log, wherein the first log and the second log are multiple segmented logs obtained by truncating the same log; it concatenates the first log and the second log according to the segment sequence number of the first log and the second log to obtain a complete target log; and it sends the complete target log.

[0162] In this embodiment of the application, the log storage system returns the complete log obtained from the search and the complete target log obtained by splicing the segmented logs as the log search results to the client.

[0163] The specific log splicing steps can be found in the above embodiments, and will not be repeated here.

[0164] In this embodiment of the application, the client can successfully complete the storage and retrieval of long logs without being aware of the log truncation and log splicing. Thus, the storage and visual reading of long logs can be achieved without the user's awareness.

[0165] Based on the same inventive concept, this application also provides a log processing device, as shown in FIG6, which illustrates a schematic diagram of the structure of the log processing device proposed in this application embodiment. The device includes:

[0166] The first receiving module 601 is used to receive logs to be written.

[0167] Module 602 is used to determine the length of the log to be written.

[0168] The truncation module 603 is used to truncate the log to be written when the length of the log to be written is greater than a first preset length, to obtain at least two segmented logs, wherein each pair of adjacent segmented logs contains repeated characters of a second preset length; the second preset length is determined based on the length of the keywords that support retrieval.

[0169] The first adding module 604 is used to add a truncation identifier and a fragment sequence number to the at least two segments of log;

[0170] Storage module 605 is used to store at least two segmented logs carrying the truncation identifier and segment number.

[0171] Optionally, the truncation module 603 is specifically used for:

[0172] The starting position of the log to be written is taken as the starting position of the first segment log, and the ending position of the first segment log is obtained by excluding a third preset length.

[0173] Starting from the end position of the first segment log and moving towards the start position, the start position of the second segment log is obtained by pausing at a second preset length. Starting from the start position of the second segment log, the end position of the second segment log is obtained by pausing at a third preset length, and so on, until the end position of the log to be written is reached. The end position of the log to be written is then taken as the end position of the last segment log.

[0174] Based on the start and end positions of each segmented log, at least two segmented logs are obtained.

[0175] Optionally, the device further includes:

[0176] The second receiving module is used to receive log search requests, wherein the log search requests carry search keywords;

[0177] The search module is used to search for and obtain the first log associated with the search keywords;

[0178] The acquisition module is used to acquire a second log related to the first log when the first log carries a truncation identifier. The first log and the second log are multiple segmented logs obtained by truncating the same log.

[0179] The splicing module is used to splice the first log and the second log according to the segment sequence number of the first log and the second log to obtain the complete target log;

[0180] The sending module is used to send the complete target log.

[0181] Optionally, the splicing module is specifically used for:

[0182] The order of each segment log is obtained by sorting the segment numbers of the first and second logs.

[0183] The second preset length is removed from the end of each segment log from the first to the second to last segment log; multiple trimmed segment logs are obtained.

[0184] The multiple cropped log segments and the last log segment are concatenated according to the segment number to obtain the complete target log.

[0185] Optionally, the device further includes:

[0186] The second adding module is used to add digital digests to the at least two segmented log segments;

[0187] The verification module is used to verify the complete target log based on the digital digest;

[0188] The verification module is specifically used for:

[0189] Send the complete target log that has passed verification.

[0190] Optionally, the device further includes:

[0191] The third receiving module is used to receive the length configuration information of the keywords that support retrieval;

[0192] The determining module is used to determine a second preset length based on the length of the keywords that support retrieval;

[0193] The third adding module is used to add a second preset length configuration information to the at least two segmented logs.

[0194] Based on the same inventive concept, this application also provides a log processing device, as shown in FIG7, which illustrates a schematic diagram of the structure of the log processing device proposed in this application embodiment. The device includes:

[0195] The first sending module 701 is used to send length configuration information of keywords that support retrieval, wherein the length configuration information is used to determine a second preset length;

[0196] The second sending module 702 is used to send a log storage request to the log storage system, wherein the log storage request carries logs to be written.

[0197] The log storage system is used to truncate the log to be written when the log to be written is greater than a first preset length, to obtain at least two segmented logs, add a truncation identifier and a segment number to the at least two segmented logs, and store the at least two segmented logs carrying the truncation identifier and the segment number, wherein each pair of adjacent segmented logs contains repeated characters of the second preset length.

[0198] Optionally, the device further includes:

[0199] The third sending module is used to send a log search request to the log storage system, and the log search request carries search keywords.

[0200] The receiving module is used to receive log search results.

[0201] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps in the log processing method as described in any of the above embodiments.

[0202] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the log processing method described in any of the above embodiments.

[0203] Based on the same inventive concept, embodiments of this application provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the log processing method described in any of the above embodiments.

[0204] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0205] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0207] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0208] These computer program instructions may also be loaded onto a computer or other programmable terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0209] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0210] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0211] The above provides a detailed description of a log processing method provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A log processing method, the method being applied to a log storage system, the method comprising: Receive logs to be written; Determine the length of the log to be written; If the length of the log to be written is greater than the first preset length, the log to be written is truncated to obtain at least two segmented logs, and each pair of adjacent segmented logs contains repeated characters of a second preset length; the second preset length is determined based on the length of the keywords that support retrieval; Add a truncation identifier and a segment number to the at least two segmented log segments; Store at least two segmented log segments carrying the truncation identifier and segment number.

2. The log processing method according to claim 1, wherein the log to be written is truncated to obtain at least two segmented log segments, including: The starting position of the log to be written is taken as the starting position of the first segment log, and the ending position of the first segment log is obtained by excluding a third preset length. Starting from the end position of the first segment log and moving towards the start position, the start position of the second segment log is obtained by pausing at a second preset length. Starting from the start position of the second segment log, the end position of the second segment log is obtained by pausing at a third preset length, and so on, until the end position of the log to be written is reached. The end position of the log to be written is then taken as the end position of the last segment log. Based on the start and end positions of each segmented log, at least two segmented logs are obtained.

3. The log processing method according to claim 2, wherein both the second preset length and the third preset length are less than the first preset length.

4. The log processing method according to claim 1, wherein the method further comprises: Receive a log search request, the log search request carrying search keywords; The first log associated with the search keyword is obtained by searching based on the search keyword; If the first log carries a truncation identifier, obtain the second log related to the first log. The first log and the second log are multiple segmented logs obtained by truncating the same log. Based on the segment sequence numbers of the first log and the second log, the first log and the second log are concatenated to obtain the complete target log; Send the complete target log.

5. The log processing method according to claim 4, wherein concatenating the first log and the second log according to the segment sequence numbers of the first log and the second log to obtain a complete target log includes: The order of each segment log is obtained by sorting the segment numbers of the first and second logs. The second preset length is removed from the end of each segment log from the first to the second to last segment log; Multiple cropped segmented logs were obtained; The multiple cropped log segments and the last log segment are concatenated according to the segment number to obtain the complete target log.

6. The log processing method according to claim 4, wherein after adding truncation identifiers and fragment numbers to the at least two segmented logs, the method further includes: Add a digital digest to the at least two segmented logs; After obtaining the complete target log, the method further includes: The complete target log is verified based on the digital digest; Send the complete target log, including: Send the complete target log that has passed verification.

7. The log processing method according to any one of claims 1-6, wherein before truncating the log to be written, the method further comprises: Receive the length configuration information of the keywords that support retrieval; The second preset length is determined based on the length of the keywords that support retrieval; After adding truncation identifiers and fragment numbers to the at least two segmented log segments, the process also includes: Add a second preset length configuration information to the at least two segmented logs.

8. A log processing method, the method being applied to a client, the method comprising: Send the length configuration information of the keywords that support retrieval, the length configuration information being used to determine the second preset length; Send a log storage request to the log storage system, the log storage request carrying the logs to be written; The log storage system is used to truncate the log to be written when the log to be written is greater than a first preset length, to obtain at least two segmented logs, add a truncation identifier and a segment number to the at least two segmented logs, and store the at least two segmented logs carrying the truncation identifier and the segment number, wherein each pair of adjacent segmented logs contains repeated characters of the second preset length.

9. A log processing apparatus, the apparatus being used in a log storage system, the apparatus comprising: The first receiving module is used to receive logs to be written. The judgment module is used to determine the length of the log to be written; The truncation module is used to truncate the log to be written when the length of the log to be written is greater than a first preset length, to obtain at least two segmented logs, wherein each pair of adjacent segmented logs contains repeated characters of a second preset length; the second preset length is determined based on the length of the keywords that support retrieval. The first adding module is used to add truncation identifiers and fragment numbers to the at least two segments of log; A storage module is used to store at least two segmented logs carrying the truncation identifier and segment number.

10. A log processing apparatus, the apparatus being applied to a client, the apparatus comprising: The first sending module is used to send length configuration information of keywords that support retrieval, wherein the length configuration information is used to determine a second preset length; The second sending module is used to send a log storage request to the log storage system, wherein the log storage request carries the log to be written; The log storage system is used to truncate the log to be written when the log to be written is greater than a first preset length, to obtain at least two segmented logs, add a truncation identifier and a segment number to the at least two segmented logs, and store the at least two segmented logs carrying the truncation identifier and the segment number, wherein each pair of adjacent segmented logs contains repeated characters of the second preset length.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the log processing method of any one of claims 1-7 or the log processing method of claim 8.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the log processing method of any one of claims 1-7 or the log processing method of claim 8.

13. A computer program product comprising a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the steps of the log processing method of any one of claims 1-7 or the log processing method of claim 8.

Images