Log processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a log processing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of data processing. Under the condition that the length of the to-be-written log is a long log exceeding a retrieval limit length, the to-be-written log needs to be stored after being cut off, repeated bytes with a second preset length exist between every two adjacent segmented logs in each cut-off log, and the to-be-written log is written on the basis of a coincident part between the adjacent segmented logs. At least one segmented log contains complete key information, so that the segmented logs after cutting can support retrieval through keywords. When the user reads the long log, the log storage system can retrieve the corresponding segmented log based on the keyword and obtain the related segmented log, and the plurality of segmented logs are spliced and restored into the long log to be returned to the user. In order to solve the problem that the long log can only be stored but cannot be searched, the invention provides an end-to-end user perception-free solution from writing to visual reading.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a log processing method, device, electronic device, storage medium, and program product. Background Art

[0002] Logging is a very important practice in software development. Logging is like taking notes on the software running process. It records various events that occur during the software running process so that technical personnel can better understand the behavior of the software, troubleshoot problems and optimize performance.

[0003] Different programming languages ​​have different log frameworks, such as Java's Log4j, Logback, Python's logging, etc. Logs can be processed in the middle or directly structured and stored in a storage engine that supports full-text search. Taking the storage engine as Elasticsearch as an example, in order to accurately match when searching logs, the Elasticsearch keyword type is generally used to store logs. However, the keyword type has a length limit. Even if the length limit is greatly relaxed, it is difficult to predict how long the user will record. Logs longer than the limit can only be stored and cannot be filtered out by retrieving other fields of the structured log. More importantly, logs longer than the keyword limit cannot be indexed and cannot be obtained by keyword search. Summary of the invention

[0004] The embodiments of the present invention provide a log processing method, device, electronic device, storage medium and program product.

[0005] A first aspect of an embodiment of the present invention provides a log processing method, characterized in that the method includes: Receive logs to be written; Determine the length of the log to be written; When the length of the log to be written is greater than the first preset length, the log to be written is truncated to obtain at least two segmented logs, and there are repeated characters of a second preset length between each two adjacent segmented logs; the second preset length is determined according to the length of the keyword supported for retrieval; Adding a truncation identifier and a fragment sequence number to the at least two segmented logs; At least two segment logs carrying the truncation identifier and the segment sequence number are stored.

[0006] Optionally, the log to be written is truncated to obtain at least two segmented logs, including: The starting position of the log to be written is used as the starting position of the first segmented log, and the ending position of the first segmented log is obtained by spacing the log by a third preset length; Starting from the end position of the first segmented log toward the starting position, the start position of the second segmented log is obtained by intervals of a second preset length; starting from the start position of the second segmented log, the end position of the second segmented log is obtained by intervals of a third preset length, and so on, until the end position of the log to be written is reached, and the end position of the log to be written is taken as the end position of the last segmented log; At least two segmented logs are obtained according to the starting position and the ending position of each segmented log.

[0007] Optionally, the method further comprises: receiving a log search request, wherein the log search request carries a search keyword; Searching according to the search keyword to obtain a first log associated with the search keyword; In a case where the first log carries a truncation identifier, obtaining a second log related to the first log, where the first log and the second log are multi-segmented logs obtained by truncating the same log; splicing the first log and the second log according to the fragment sequence numbers of the first log and the second log to obtain a complete target log; Send the complete target log.

[0008] Optionally, according to the fragment sequence numbers of the first log and the second log, the first log and the second log are concatenated to obtain a complete target log, including: According to the fragment sequence numbers of the first log and the second log, sorting is performed to obtain the order of each segment log; The second preset length is trimmed from the end of each segment log from the first to the second to last segment logs to obtain a plurality of trimmed segment logs; The multiple clipped segment logs and the last segment log are spliced ​​according to the fragment sequence numbers to obtain the complete target log.

[0009] Optionally, after adding the truncation identifier and the fragment sequence number to the at least two segmented logs, the method further comprises: adding a digital summary to the at least two segmented logs; After obtaining the complete target log, the method further includes: Verifying the complete target log according to the digital summary; Send the complete target log, including: Sends a complete target log that has passed verification.

[0010] Optionally, before truncating the log to be written, the method further includes: Receive length configuration information of keywords supported for retrieval; Determine a second preset length according to the length of the keyword supporting retrieval; After adding the truncation identifier and the fragment sequence number to the at least two segmented logs, the method further includes: Add second preset length configuration information to the at least two segmented logs.

[0011] A second aspect of an embodiment of the present invention provides a log processing method, the method comprising: Sending length configuration information of a keyword supported for retrieval, wherein the length configuration information is used to determine a second preset length; Sending a log storage request to a log storage system, wherein the log storage request carries a log to be written; The log storage system is used to truncate the log to be written to obtain at least two segmented logs when the log to be written is greater than a first preset length, and add a truncation identifier and a fragment sequence number to the at least two segmented logs; store the at least two segmented logs carrying the truncation identifier and the fragment sequence number, and each two adjacent segmented logs have repeated characters of the second preset length.

[0012] A third aspect of an embodiment of the present invention provides a log processing device, the device comprising: A first receiving module, used for receiving a log to be written; A judgment module is used to judge the length of the log to be written; A truncation module, configured to truncate the log to be written to obtain at least two segmented logs when the length of the log to be written is greater than a first preset length, wherein there are repeated characters of a second preset length between each two adjacent segmented logs; the second preset length is determined according to the length of the keyword supported for retrieval; A first adding module, configured to add a truncation identifier and a fragment sequence number to the at least two segmented logs; The storage module is used to store at least two segment logs carrying the truncation identifier and the segment sequence number.

[0013] Optionally, the truncation module is specifically used to: The starting position of the log to be written is used as the starting position of the first segmented log, and the ending position of the first segmented log is obtained by spacing the log by a third preset length; Starting from the end position of the first segmented log toward the starting position, the start position of the second segmented log is obtained by intervals of a second preset length; starting from the start position of the second segmented log, the end position of the second segmented log is obtained by intervals of a third preset length, and so on, until the end position of the log to be written is reached, and the end position of the log to be written is taken as the end position of the last segmented log; At least two segmented logs are obtained according to the starting position and the ending position of each segmented log.

[0014] Optionally, the device further comprises: A second receiving module is used to receive a log search request, where the log search request carries a search keyword; A search module, configured to search for a first log associated with the search keyword according to the search keyword; an acquisition module, configured to acquire, when the first log carries a truncation identifier, a second log related to the first log, wherein the first log and the second log are multi-segment logs obtained by truncating the same log; a splicing module, configured to splice the first log and the second log according to the fragment sequence numbers of the first log and the second log to obtain a complete target log; A sending module is used to send the complete target log.

[0015] Optionally, the splicing module is specifically used for: According to the fragment sequence numbers of the first log and the second log, sorting is performed to obtain the order of each segment log; The second preset length is trimmed from the end of each segment log from the first to the second to last segment logs to obtain a plurality of trimmed segment logs; The multiple clipped segment logs and the last segment log are spliced ​​according to the fragment sequence numbers to obtain the complete target log.

[0016] Optionally, the device further comprises: A second adding module, configured to add a digital summary to the at least two segmented logs; A verification module, used for verifying the complete target log according to the digital summary; The verification module is specifically used for: Sends a complete target log that has passed verification.

[0017] Optionally, the device further comprises: A third receiving module is used to receive length configuration information of keywords supported for retrieval; A determination module, configured to determine a second preset length according to the length of the keyword supporting retrieval; The third adding module is used to add second preset length configuration information to the at least two segmented logs.

[0018] A fourth aspect of an embodiment of the present invention provides a log processing device, the device comprising: A first sending module, used for sending length configuration information of a keyword supported for retrieval, wherein the length configuration information is used for determining a second preset length; A second sending module, configured to send a log storage request to a log storage system, wherein the log storage request carries a log to be written; The log storage system is used to truncate the log to be written to obtain at least two segmented logs when the log to be written is greater than a first preset length, and add a truncation identifier and a fragment sequence number to the at least two segmented logs; store the at least two segmented logs carrying the truncation identifier and the fragment sequence number, and each two adjacent segmented logs have repeated characters of the second preset length.

[0019] A fifth aspect of an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the log processing method as described in the first aspect or the second aspect of the present invention.

[0020] A sixth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log processing method as described in the first aspect or the second aspect of the present invention.

[0021] A seventh aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which is implemented by a processor to implement the steps in the log processing method described in the first aspect or the second aspect of the present invention.

[0022] In an embodiment of the present invention, the length of the log to be written is greater than the first preset length, indicating that the log to be written is a long log that exceeds the retrieval limit length and needs to be truncated before storage. In an embodiment of the present invention, in each segmented log after truncation, there are repeated bytes of the second preset length between each adjacent segmented log. Therefore, in an embodiment of the present invention, based on the overlap of the second preset length between each adjacent segmented log, at least one segmented log contains complete key information, so that the multiple segmented logs after clipping can support retrieval by keywords. In a common truncation scheme, assuming that the truncated part is just a mobile phone number or user ID, when the user searches using the mobile phone number or user ID, the truncated log cannot be searched. In an embodiment of the invention, due to the overlap of the second preset length between multiple segmented logs, even if the mobile phone number or user ID is truncated in the previous segmented log, in the next segmented log, the mobile phone number or user ID is still complete and can effectively support retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0024] Figure 1 is a flow chart of the steps of the log processing method provided by an embodiment of the present invention; Figure 2 Schematic diagram of segmented logs in the log processing method provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of the overall flow of the log processing method provided by an embodiment of the present invention; Figure 4 It is a log search flow diagram of the log processing method provided by an embodiment of the present invention; Figure 5 is a flow chart of the steps of the log processing method provided by an embodiment of the present invention; Figure 6 is a structural diagram of a log processing device provided by an embodiment of the present invention; Figure 7 It is a structural diagram of a log processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Regarding the problem of how to store long logs, one solution adopted in the related art is to truncate the long logs. In this solution, if the user's more important information is truncated, it will not be searchable. When the user searches the log, accurate matching cannot be achieved. For the user, the log seems not to be stored, and the log loses its function.

[0027] One solution used in related technologies is to use the text type to store long logs. This solution will cause the log storage system to be unable to accurately match the logs. When users search based on keywords, many irrelevant results will appear, affecting the search experience.

[0028] Based on this, the present invention proposes to truncate long logs into log segments shorter than the keyword limit for storage, and retain the overlapping parts in the multiple segmented logs obtained by truncation based on the keyword length supported for retrieval. This can solve the problem that long logs cannot be retrieved when stored using keywords, which is better than direct truncation and using the text type to store the log body.

[0029] Specifically, the embodiment of the present invention proposes a log processing method, such as Figure 1 As shown, it shows a flow chart of the steps of the log processing method proposed in an embodiment of the present invention, and the method includes the following steps: S101, receiving a log to be written.

[0030] S102, determining the length of the log to be written.

[0031] S103: When the length of the log to be written is greater than the first preset length, the log to be written is truncated to obtain at least two segmented logs, and each of two adjacent segmented logs has repeated characters of a second preset length.

[0032] S104: Add a truncation identifier and a fragment sequence number to the at least two segmented logs.

[0033] S105: Store at least two segment logs carrying the truncation identifier and the segment sequence number.

[0034] In an embodiment of the present invention, the method can be applied to a log storage system, for example, Elasticsearch. Elasticsearch is an open source distributed search and analysis engine. It is based on the Apache Lucene library and is written in Java. Elasticsearch can efficiently store, search and analyze massive amounts of data, and is particularly good at processing unstructured data, such as text, logs, geospatial data, etc.

[0035] In the embodiment of the present invention, the user may initiate a log storage request through a write gateway of the log storage system, and carry the log to be written in the request.

[0036] In an embodiment of the present invention, the log to be written may include a text file of a series of events recorded by the software during operation. These events may include: error information: error prompts generated when the software encounters problems during operation; warning information: potential problems or abnormal situations; debugging information: detailed records used by developers to track the execution process of the program; user operations: operations performed by users in the software, such as logging in, clicking buttons, etc.; system information: system status when the software is running, such as memory usage, CPU occupancy, etc.

[0037] In the embodiment of the present invention, the character length of the log to be written may be judged. When the length of the log to be written is greater than a first preset length, it is determined to be a long log and needs to be truncated before being stored.

[0038] In the embodiment of the present invention, the first preset length is the search limit length. Long logs exceeding the search limit length cannot be indexed after storage and cannot be filtered out by searching other fields of the structured log. For example, the length can be a keyword limit. Specifically, the first preset length can be, for example, 1024 characters.

[0039] Therefore, the embodiment of the present invention proposes to adopt the truncation step of step S103 to retain the overlapping parts for every two adjacent segment logs, so as to avoid important information being truncated and causing subsequent retrieval failure.

[0040] Specifically, in the embodiment of the present invention, step S103 may include the following sub-steps: S1031, taking the starting position of the log to be written as the starting position of the first segmented log, and obtaining the ending position of the first segmented log by a third preset length.

[0041] S1032, starting from the end position of the first segmented log toward the starting position, the starting position of the second segmented log is obtained at an interval of a second preset length, starting from the starting position of the second segmented log, the end position of the second segmented log is obtained at an interval of a third preset length, and so on, until the end position of the log to be written is reached, and the end position of the log to be written is used as the end position of the last segmented log.

[0042] S1033 obtains at least two segment logs according to the start position and the end position of each segment log.

[0043] For ease of understanding, the embodiment of the present invention provides a segmented log schematic diagram, such as Figure 2As shown, 0 represents the starting position of the log to be written, M represents the starting position of the second segmented log, N represents the ending position of the first segmented log, ~ represents that the log has not ended, and the subsequent segmentation method is the same as the second segment method, and NM is determined according to the length of the keyword supported for retrieval. In the embodiment of the present invention, both M and N can be parameterized by the user.

[0044] In an embodiment of the present invention, after determining the starting position and ending position of each segmented log, the content to be written to the corresponding position of the log can be copied based on the starting position and ending position of each segmented log to obtain each segmented log, and each segmented log is stored in a log storage system.

[0045] In the embodiment of the present invention, during the truncation process, the log can be truncated according to the third preset length to obtain the first segmented log, and then the segmented log of the second preset length can be copied from the end position of the first segmented log to the starting position, and the segmented log can be added to the starting position of the remaining log after truncating the first segmented log according to the content order of the original log to be written, and then the log can be truncated from the starting position of the remaining position according to the third preset length to obtain the second segmented log, and so on, until the length of the remaining log does not exceed the third preset length. In this way, at least two segmented logs can be truncated, and there are repeated characters of the second preset length in each adjacent two segmented logs.

[0046] In the embodiment of the present invention, no specific restriction is imposed on the truncation strategy during the truncation process, and only a restriction is imposed on storing repeated characters of a second preset length between each two adjacent segmented logs in at least two segmented logs obtained after truncation.

[0047] In the embodiment of the present invention, the third preset length (such as the N value in the above example) is smaller than the first preset length, so as to truncate the long log into segmented logs that do not exceed the retrieval limit length (the first preset length), so that the stored segmented logs can be retrieved. For example, if the first preset length is 1024 characters, the third preset length may be 1000 characters.

[0048] In the embodiment of the present invention, the third preset length may be specified by the user according to actual needs when initiating a log storage request.

[0049] In an embodiment of the present invention, the second preset length may be the length of a keyword that supports retrieval, which is generally set to one tenth of the first preset length. It may also be specified by the user according to his or her actual needs when initiating a log storage request. For example, if the user needs to retrieve the log by an 11-character user mobile phone number later, the second preset length may be set to 11 characters. Then, at least one of the multiple segmented logs obtained after the long log is truncated contains the complete user mobile phone number, and accurate retrieval of the log can be achieved.

[0050] In the embodiment of the present invention, in order to achieve smooth search and splicing of the segmented logs later, the embodiment of the present invention also proposes to add a truncation identifier and a fragment sequence number to the segmented logs.

[0051] Specifically, in the embodiment of the present invention, a truncation identifier may be added to the segmented log in a specific field, for example, a UUID may be used for filling, and then all segmented logs included in the long log may be found based on the truncation identifier.

[0052] In the embodiment of the present invention, for a plurality of segmented logs obtained by truncating a long log, the truncation identifiers thereof are unique.

[0053] In the embodiment of the present invention, the truncation identifier can indicate that the log segment stored in the log storage system is a segmented log, and all segmented logs of the same long log can also be found based on the truncation identifier.

[0054] In the embodiment of the present invention, a segment sequence number may be added to the segment log in a specific field, so that the segment logs can be subsequently spliced ​​in sequence according to the segment sequence numbers of the segment logs to obtain a complete long log.

[0055] It should be noted that, in the embodiment of the present invention, although characters are repeated between every two adjacent segmented logs in the segmented logs stored in the log storage system, in the embodiment of the present invention, the length limit supported for retrieval can be enlarged, for example: 1024 characters, then the number of long logs exceeding the retrieval limit length is relatively small, and corresponding to the overall log storage system, it will not increase excessive waste of storage resources.

[0056] In the embodiment of the present invention, after a user writes a log through a write gateway of the log storage system and completes the storage of the log, the user can also read the stored log through a read gateway of the log storage system.

[0057] Specifically, in the embodiment of the present invention, the method further includes the following steps: S106: Receive a log search request, where the log search request carries a search keyword.

[0058] S107: Search according to the search keyword to obtain a first log associated with the search keyword.

[0059] S108: When the first log carries a truncation identifier, obtain a second log related to the first log, where the first log and the second log are multi-segment logs obtained by truncating the same log.

[0060] S109: splice the first log and the second log according to the fragment sequence numbers of the first log and the second log to obtain a complete target log.

[0061] S110: Send the complete target log.

[0062] Specifically, in an embodiment of the present invention, a user can initiate a log search request through a read gateway of a log storage system, and carry a search keyword in the search request. The read gateway can then search the log storage system for all logs associated with the search keyword, which may include complete logs and segmented logs that have been truncated and stored.

[0063] In the embodiment of the present invention, when the searched log carries a truncation identifier, indicating that the log is a segmented log of a long log, other segmented logs related to the log can be found based on the truncation identifier. These segmented logs can be spliced ​​into a complete log.

[0064] In the embodiment of the present invention, after the search and splicing are completed, all complete target logs associated with the search keyword can be returned to the user, so that the user can accurately search and read the long log by searching the keyword.

[0065] For ease of understanding, the following Figure 3 The overall writing and reading process of the log processing method provided by the embodiment of the present invention is described. Figure 3 The overall flow chart of the log processing method provided by the embodiment of the present invention is shown as follows: Figure 3 As shown, in the embodiment of the present invention, the user can write a long log to the log storage system (Elasticsearch is used as an example in the figure) through the write gateway, and the write gateway can truncate the long log to obtain at least two segmented logs, and each adjacent segmented log has a second preset length of repeated characters, so as to write the long log segments into the log storage system. In this process, the user is not aware of this, and the storage of the long log is successfully completed for the user.

[0066] When it is necessary to search and read logs, users can initiate a search request to the log storage system through the reading gateway based on the search keyword. The reading gateway then searches the log storage system for all logs related to the search keyword based on the search keyword. If there is a log with a truncated identifier, all segmented logs related to the log can be obtained based on the truncated identifier (such as UUID) of the log, and multiple segmented logs can be spliced ​​according to the splicing rules to obtain the complete target log, and finally all complete target logs associated with the search keyword are returned to the user.

[0067] Therefore, in the embodiment of the present invention, when a user reads a long log, the log storage system can retrieve the corresponding segmented log based on the keyword and obtain other segmented logs related to the segmented log, splice and restore multiple segmented logs into a long log, and return the long log to the user, thereby realizing visual reading of the long log without the user's perception.

[0068] In the embodiment of the present invention, in the above step S109, the splicing process of each segment log may specifically include the following sub-steps: S1091: Sort the fragment numbers of the first log and the second log to obtain the order of each segment log.

[0069] S1092: trimming and removing a second preset length from the end of each segment log from the first to the second to last segment logs to obtain a plurality of trimmed segment logs.

[0070] S1093: splice the multiple cropped segment logs and the last segment log according to the segment sequence numbers to obtain the complete target log.

[0071] Specifically, in the embodiment of the present invention, the content of the second preset length may be cut off from the end of all logs except the last segmented log to avoid duplication.

[0072] In the embodiment of the present invention, the writing gateway may also add a digital summary to the segmented log to facilitate the subsequent content verification of the spliced ​​log. Specifically, in step S104, a digital summary may be added to the at least two segmented logs obtained by truncation. In the embodiment of the present invention, the digital summary is generated based on the log content of the original log to be written.

[0073] In an embodiment of the present invention, after obtaining all segmented logs of the same long log, the reading gateway can also obtain a digital summary of the long log from a preset field. After step S1093, the method further includes: verifying the complete target log according to the digital summary.

[0074] In the embodiment of the present invention, in step S110, the complete target log that has passed the verification is sent. For the spliced ​​log that has failed the verification, a prompt tag indicating that the integrity verification has failed can be added, and the spliced ​​log carrying the prompt tag is returned.

[0075] In the embodiment of the present invention, the user can also configure the second preset length. Specifically, continue to combine Figure 2 To illustrate, the user can configure the value of NM, and the log storage system can determine the values ​​of M and N, and then determine the starting position and the ending position of each segmented log, and complete the truncation of the long log.

[0076] Specifically, in the embodiment of the present invention, before step S103, the method may further include: S1, receiving the length configuration information of the keyword supported for retrieval.

[0077] S2: Determine a second preset length according to the length of the keyword supported for retrieval.

[0078] In this case, in step S104, after adding the truncation identifier and the fragment sequence number to the segmented log, the second preset length configuration information needs to be added to the segmented log. Thus, the reading gateway can complete the splicing of the segmented logs according to the found second preset length configuration information of the segmented logs.

[0079] For ease of understanding, the following Figure 4 The reading process of the log processing method provided by the embodiment of the present invention is described. Figure 4 A log search flow diagram of a log processing method provided by an embodiment of the present invention is shown. Figure 4 As shown, in an embodiment of the present invention, a user can initiate a log search request to the read gateway of the log storage system by searching for a keyword K, and the read gateway can search for all fragments containing K from the log storage system (Elasticsearch is taken as an example in the figure), and the log storage system returns all the searched fragments containing K to the read gateway. Among these fragments, if there is a log fragment carrying a truncated identifier, the read gateway further searches for the log fragment containing the truncated identifier from all the log fragments stored in the log storage system based on the truncated identifier (UUID).

[0080] Specifically, in an embodiment of the present invention, the fragment containing K may contain multiple log fragments carrying truncation identifiers. Since the truncation identifier corresponding to each long log is unique after being truncated, in an embodiment of the present invention, the reading gateway can concurrently search for all log fragments from the log storage system based on each unique truncation identifier.

[0081] For a certain log segment that contains K and carries a truncation identifier, after searching for all segments that contain the truncation identifier, the segments can be sorted based on the segment serial numbers carried by each log segment. For example, the segment serial numbers can be: S1, S2, S3...SN. In the embodiment of the present invention, for log segments with subsequent serial numbers (that is, all log segments except the last log segment), the ends of these log segments can be trimmed according to the second preset length (NM) determined by the pre-parameterized configuration, and the trimmed log segments S1', S2', S3'...SN can be spliced. Then, the spliced ​​log can be verified based on the digital summary, and all complete logs containing K (including: the spliced ​​complete target log and the original log that has not been truncated) are returned to the user.

[0082] Compared with ordinary truncation, in the embodiment of the present invention, a second preset length that can support parameterized configuration is added during the truncation process of the log. Based on the overlap of the second preset lengths between adjacent segmented logs, the multiple segmented logs after trimming can still support retrieval. In the ordinary truncation scheme, assuming that the truncated part is just a mobile phone number or user ID, when the user searches using the mobile phone number or user ID, the truncated log cannot be searched. In the embodiment of the invention, due to the overlap of the second preset lengths between multiple segmented logs, even if the mobile phone number or user ID is truncated in the previous segmented log, in the next segmented log, the mobile phone number or user ID is still complete and can effectively support retrieval.

[0083] In addition, in the embodiment of the present invention, the second preset length supports parameterized configuration, and the second preset length configuration information can be added to the specified field of the segmented log obtained by truncation, such as adding a fixed number of bits after the log segment, for example, a bitmap method can be used, so that the user can parameterize the second preset length in the long log truncation process according to actual needs, and the reading gateway can still accurately complete the search and splicing of the segmented logs when the second preset length is constantly changing. Therefore, the embodiment of the present invention provides an end-to-end user-imperceptible solution from writing to visual reading to solve the problem that long logs can only be stored but not searched.

[0084] Based on the same inventive concept, the embodiment of the present invention also provides a log processing method, such as Figure 5 As shown, it shows a flow chart of the steps of the log processing method provided by an embodiment of the present invention. Specifically, the method includes the following steps: S501, sending length configuration information of a keyword supported for retrieval, where the length configuration information is used to determine a second preset length.

[0085] S502: Send a log storage request to a log storage system, where the log storage request carries a log to be written.

[0086] The log storage system is used to truncate the log to be written to obtain at least two segmented logs when the log to be written is greater than a first preset length, and add a truncation identifier and a fragment sequence number to the at least two segmented logs; store the at least two segmented logs carrying the truncation identifier and the fragment sequence number, and each two adjacent segmented logs have repeated characters of the second preset length.

[0087] In an embodiment of the present invention, the log processing method is used for a client, and a user can send the length configuration information of the keyword supported for retrieval to the log storage system based on the client. When the length of the log uploaded by the user through the client exceeds the retrieval limit length, the log storage system can determine the second preset length based on the length configuration information of the keyword to be retrieved, and then truncate the long log into segmented logs with a length less than the retrieval limit length based on the second preset length, and there are repeated characters of the second preset length in each adjacent segmented log. Therefore, in an embodiment of the invention, a long log can be stored in segments and pointed out for retrieval, and because the second preset lengths between multiple segmented logs overlap, even if the key information is truncated in the previous segmented log, the key information is still complete in the next segmented log, and can effectively support retrieval.

[0088] In an embodiment of the present invention, the second preset length can be configured by the user by sending length configuration information of the keyword supported for retrieval through the client, so that the user can parameterize the second preset length according to actual needs. Of course, in an embodiment of the present invention, the second preset length can also be configured in advance by a technician, and the user does not need to configure the second preset length every time the last log is logged.

[0089] In the embodiment of the present invention, the specific log truncation steps can refer to the above embodiment and will not be described in detail here.

[0090] In the embodiment of the present invention, the method may further include the following steps: S503: Send a log search request to the log storage system, where the log search request carries a search keyword.

[0091] S504: Receive log search results.

[0092] The log storage system is used to search for a first log associated with the search keyword according to the search keyword; when the first log carries a truncation identifier, obtain a second log related to the first log, wherein the first log and the second log are multi-segment segmented logs obtained by truncating the same log; according to the fragment sequence numbers of the first log and the second log, splice the first log and the second log to obtain a complete target log; and send the complete target log.

[0093] In the embodiment of the present invention, the log storage system returns the complete log obtained by searching and the complete target log obtained by splicing the segmented logs to the client as log search results.

[0094] The specific log concatenation steps can be referred to the above embodiment and will not be described in detail here.

[0095] In the embodiment of the present invention, for the client, the storage and search of long logs can be completed smoothly, and the log truncation and log splicing therein can be unaware of it, so that the storage and visual reading of long logs can be achieved without the user's awareness.

[0096] Based on the same inventive concept, the embodiment of the present invention also provides a log processing device, such as Figure 6 As shown, it shows a schematic diagram of the structure of a log processing device proposed in an embodiment of the present invention, and the device includes: A first receiving module 601 is used to receive a log to be written; A determination module 602 is used to determine the length of the log to be written; The truncation module 603 is used to truncate the log to be written to obtain at least two segmented logs when the length of the log to be written is greater than a first preset length, and each of the two adjacent segmented logs has repeated characters of a second preset length; the second preset length is determined according to the length of the keyword supported for retrieval; A first adding module 604 is used to add a truncation identifier and a fragment sequence number to the at least two segmented logs; The storage module 605 is used to store at least two segment logs carrying the truncation identifier and the segment sequence number.

[0097] Optionally, the truncation module 603 is specifically configured to: The starting position of the log to be written is used as the starting position of the first segmented log, and the ending position of the first segmented log is obtained by spacing the log by a third preset length; Starting from the end position of the first segmented log toward the starting position, the start position of the second segmented log is obtained by intervals of a second preset length; starting from the start position of the second segmented log, the end position of the second segmented log is obtained by intervals of a third preset length, and so on, until the end position of the log to be written is reached, and the end position of the log to be written is taken as the end position of the last segmented log; At least two segmented logs are obtained according to the starting position and the ending position of each segmented log.

[0098] Optionally, the device further comprises: A second receiving module is used to receive a log search request, where the log search request carries a search keyword; A search module, configured to search for a first log associated with the search keyword according to the search keyword; an acquisition module, configured to acquire, when the first log carries a truncation identifier, a second log related to the first log, wherein the first log and the second log are multi-segment logs obtained by truncating the same log; a splicing module, configured to splice the first log and the second log according to the fragment sequence numbers of the first log and the second log to obtain a complete target log; A sending module is used to send the complete target log.

[0099] Optionally, the splicing module is specifically used for: According to the fragment sequence numbers of the first log and the second log, sorting is performed to obtain the order of each segment log; The second preset length is trimmed from the end of each segment log from the first to the second to last segment logs to obtain a plurality of trimmed segment logs; The multiple clipped segment logs and the last segment log are spliced ​​according to the fragment sequence numbers to obtain the complete target log.

[0100] Optionally, the device further comprises: A second adding module, configured to add a digital summary to the at least two segmented logs; A verification module, used for verifying the complete target log according to the digital summary; The verification module is specifically used for: Sends a complete target log that has passed verification.

[0101] Optionally, the device further comprises: A third receiving module is used to receive length configuration information of keywords supported for retrieval; A determination module, configured to determine a second preset length according to the length of the keyword supporting retrieval; The third adding module is used to add second preset length configuration information to the at least two segmented logs.

[0102] Based on the same inventive concept, the embodiment of the present invention also provides a log processing device, such as Figure 7 As shown, it shows a schematic diagram of the structure of a log processing device proposed in an embodiment of the present invention, and the device includes: A first sending module 701, configured to send length configuration information of a keyword supported for retrieval, wherein the length configuration information is used to determine a second preset length; A second sending module 702 is used to send a log storage request to a log storage system, wherein the log storage request carries a log to be written; The log storage system is used to truncate the log to be written to obtain at least two segmented logs when the log to be written is greater than a first preset length, and add a truncation identifier and a fragment sequence number to the at least two segmented logs; store the at least two segmented logs carrying the truncation identifier and the fragment sequence number, and each two adjacent segmented logs have repeated characters of the second preset length.

[0103] Optionally, the device further comprises: The third sending module is used to send a log search request to the log storage system, where the log search request carries a search keyword.

[0104] The receiving module is used to receive log search results.

[0105] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the log processing method described in any of the above embodiments are implemented.

[0106] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the log processing method described in any of the above embodiments are implemented.

[0107] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the steps of the log processing method described in any of the above embodiments when executed by a processor.

[0108] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0109] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0110] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0114] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0115] The above is a detailed introduction to a log processing method provided by the present invention. In this article, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A log processing method, characterized in that: The method comprises: Receive logs to be written; Determine the length of the log to be written; When the length of the log to be written is greater than the first preset length, the log to be written is truncated to obtain at least two segmented logs, and there are repeated characters of a second preset length between each two adjacent segmented logs; the second preset length is determined according to the length of the keyword supported for retrieval; Adding a truncation identifier and a fragment sequence number to the at least two segmented logs; At least two segment logs carrying the truncation identifier and the segment sequence number are stored.

2. The log processing method according to claim 1, characterized in that: The log to be written is truncated to obtain at least two segmented logs, including: The starting position of the log to be written is used as the starting position of the first segmented log, and the ending position of the first segmented log is obtained by spacing by a third preset length; Starting from the end position of the first segmented log toward the starting position, the start position of the second segmented log is obtained by intervals of a second preset length; starting from the start position of the second segmented log, the end position of the second segmented log is obtained by intervals of a third preset length, and so on, until the end position of the log to be written is reached, and the end position of the log to be written is taken as the end position of the last segmented log; At least two segmented logs are obtained according to the starting position and the ending position of each segmented log.

3. The log processing method according to claim 1, characterized in that: The method further comprises: receiving a log search request, wherein the log search request carries a search keyword; Searching according to the search keyword to obtain a first log associated with the search keyword; In a case where the first log carries a truncation identifier, obtaining a second log related to the first log, where the first log and the second log are multi-segmented logs obtained by truncating the same log; splicing the first log and the second log according to the fragment sequence numbers of the first log and the second log to obtain a complete target log; Send the complete target log.

4. The log processing method according to claim 3, characterized in that: According to the fragment sequence numbers of the first log and the second log, the first log and the second log are concatenated to obtain a complete target log, including: According to the fragment sequence numbers of the first log and the second log, sorting is performed to obtain the order of each segment log; The second preset length is trimmed from the end of each segment log from the first to the second to last segment log to obtain a plurality of trimmed segment logs; The multiple clipped segment logs and the last segment log are spliced ​​according to the fragment sequence numbers to obtain the complete target log.

5. The log processing method according to claim 3, characterized in that: After adding the truncation identifier and the fragment sequence number to the at least two segmented logs, the method further includes: adding a digital summary to the at least two segmented logs; After obtaining the complete target log, the method further includes: Verifying the complete target log according to the digital summary; Send the complete target log, including: Sends a complete target log that has passed verification.

6. The log processing method according to any one of claims 1 to 5, characterized in that: Before truncating the log to be written, the method further includes: Receive length configuration information of keywords supported for retrieval; Determine a second preset length according to the length of the keyword supporting retrieval; After adding the truncation identifier and the fragment sequence number to the at least two segmented logs, the method further includes: Add second preset length configuration information to the at least two segmented logs.

7. A log processing method, characterized in that: The method comprises: Sending length configuration information of a keyword supported for retrieval, wherein the length configuration information is used to determine a second preset length; Sending a log storage request to a log storage system, wherein the log storage request carries a log to be written; The log storage system is used to truncate the log to be written to obtain at least two segmented logs when the log to be written is greater than a first preset length, and add a truncation identifier and a fragment sequence number to the at least two segmented logs; store the at least two segmented logs carrying the truncation identifier and the fragment sequence number, and each two adjacent segmented logs have repeated characters of the second preset length.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the log processing method according to any one of claims 1 to 6 or the log processing method according to claim 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the log processing method according to any one of claims 1 to 6 or the log processing method according to claim 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the log processing method described in any one of claims 1 to 6 or the steps in the log processing method described in claim 7 are implemented.

Images