Log persistent storage method and device, electronic equipment and storage medium

Abstract

The invention provides a log persistent storage method and device, electronic equipment and a storage medium, and the method comprises the steps: updating a dictionary with a word group length as a preset length, and setting an initial value of a variable as the preset length; counting the occurrence frequency of each phrase in the dictionary with the phrase length as the variable in the current log text, screening the phrases with the occurrence frequency smaller than a preset threshold value, subtracting 1 from the variable, and updating the dictionary with the phrase length as the variable based on the phrases with the occurrence frequency smaller than the preset threshold value; and repeating the steps until the value of the variable is 1, extracting the log template of the current log text based on the phrases with the occurrence frequency smaller than a preset threshold value and the phrase length being 2, and extracting variable data of the current log text based on the sample template matched with the log template of the current log text. And the variable data of the current log text and the identification information of the matched sample template are stored in the persistent storage system, so that the storage space required by the log text is reduced.

Description

Technical Field

[0001] The present invention relates to the field of log processing technology, and in particular to a log persistent storage method, device, electronic device and storage medium. Background Art

[0002] In the automated material handling system (AMHS) of a semiconductor factory, the overhead crane system is responsible for efficiently and accurately transporting key materials such as wafers between production equipment and storage equipment. In order to ensure the safe and efficient operation of the overhead crane system, log recording is essential. The logs of the overhead crane system are used to record key data such as equipment operating status, fault information, scheduling activities, and transportation routes. These logs not only help maintenance personnel troubleshoot faults and optimize scheduling strategies, but also trace abnormal events to ensure stable operation of the system. In addition, the logs can provide data support for future performance improvements and help improve the overall efficiency of the system.

[0003] However, as the wafer fab capacity increases, the overhead crane system in the AMHS runs frequently, generating a large amount of log data every day. These log data need to be persistently stored for subsequent analysis or tracing. Therefore, the massive log storage demand has become an important issue in system design, especially during a long production cycle, when storage requirements will continue to increase. In order to meet the persistent storage requirements of massive logs, it is necessary to provide an optimized solution for persistent log storage to reduce the storage pressure of logs. Summary of the invention

[0004] The present invention provides a log persistent storage method, device, electronic device and storage medium, which are used to solve the defect of high pressure of mass log storage in the prior art.

[0005] The present invention provides a log persistent storage method, comprising: Initialization step: based on the word segmentation of the current log text, updating the dictionary with a phrase length of a preset length, and setting the initial value of the variable variable to the preset length; Variable data detection step: counting the frequency of occurrence of each phrase in the dictionary whose phrase length is the variable variable in the current log text, screening the phrases whose occurrence frequency is less than a preset threshold, and updating the dictionary whose phrase length is the variable variable based on the phrases whose occurrence frequency is less than the preset threshold after decrementing the variable variable by 1; Iteration step: repeating the variable data detection step until the value of the variable variable is 1; Template extraction step: extracting a log template of the current log text based on a phrase with a length of 2 and an appearance frequency less than a preset threshold; Persistent storage step: match the log template of the current log text with each sample template in the log text library, and after obtaining the sample template matching the current log text, extract the variable data of the current log text based on the sample template matching the current log text, and save the variable data of the current log text and the identification information of the sample template matching it to the persistent storage system.

[0006] According to a method for persistent storage of logs provided by the present invention, the log template of the current log text is extracted based on a phrase with a length of 2 and an appearance frequency less than a preset threshold, including: Determine whether there are overlapping participles between the phrases of length 2 whose occurrence frequency is less than a preset threshold; If there are overlapping participles between any two phrases of length 2 whose occurrence frequency is less than a preset threshold, determining the position of the overlapping participles in the current log text as a dynamic position; The dynamic position in the current log text is filled with a preset character string to obtain a log template of the current log text.

[0007] According to a method for persistent storage of logs provided by the present invention, the updating of a dictionary with a phrase length of a preset length based on the word segmentation of the current log text specifically includes: Add the last word segment of the previous log text to the beginning of the current log text, and add the first word segment of the next log text to the end of the current log text; The word segments of the current log text are traversed based on a sliding window with a preset width, and a dictionary with a phrase length of the preset length is updated; wherein the sliding window slides for a length of one word each time.

[0008] According to a log persistent storage method provided by the present invention, the preset threshold is determined based on the following steps: Get the number of phrases corresponding to each frequency of occurrence in the dictionary with a phrase length of 2; Taking the frequency of occurrence as the independent variable and the number of phrases corresponding to each frequency of occurrence as the dependent variable, determine the first-order derivative of each dependent variable with respect to the corresponding independent variable; The mutation point of the first-order derivative of each dependent variable relative to the corresponding independent variable is determined as the preset threshold.

[0009] According to a log persistent storage method provided by the present invention, the mutation point of the first-order derivative of each dependent variable relative to the corresponding independent variable is determined based on the following method: Clustering the first-order derivative of each dependent variable with respect to the corresponding independent variable yields two clusters; The mutation point is determined based on the independent variables corresponding to each first-order derivative in the two clusters; wherein the first-order derivative corresponding to the independent variable smaller than the mutation point and the first-order derivative corresponding to the independent variable greater than the mutation point belong to different clusters.

[0010] According to a log persistent storage method provided by the present invention, matching the log template of the current log text with each sample template in the log text library to obtain the sample template matched by the current log text includes: If the log template of the current log text is completely consistent with any sample template in the log text library, then it is determined that the two match; If the matching results between the log template of the current log text and each sample template in the log text library are all mismatched, the log template of the current log text is stored in the log text library as a new sample template and the corresponding identification information is obtained.

[0011] The present invention also provides a log persistent storage device, comprising: An initialization unit, used to update a dictionary with a phrase length of a preset length based on the word segmentation of the current log text, and set an initial value of a variable variable to the preset length; A variable data detection unit, configured to execute a variable data detection step and an iteration step; wherein the variable data detection step includes counting the frequency of occurrence of each phrase in the dictionary whose phrase length is the variable variable in the current log text, screening the phrases whose occurrence frequency is less than a preset threshold, and updating the dictionary whose phrase length is the variable variable based on the phrases whose occurrence frequency is less than the preset threshold after decrementing the variable variable by 1; the iteration step includes repeating the variable data detection step until the value of the variable variable is 1; A template extraction unit, configured to extract a log template of the current log text based on a phrase with a length of 2 and an appearance frequency less than a preset threshold; A persistent storage unit is used to match the log template of the current log text with each sample template in the log text library, and after obtaining the sample template matching the current log text, extract the variable data of the current log text based on the sample template matching the current log text, and save the variable data of the current log text and the identification information of the sample template matching it in the persistent storage system.

[0012] According to a log persistent storage device provided by the present invention, the log template of the current log text is extracted based on a phrase with a length of 2 and an appearance frequency less than a preset threshold, including: Determine whether there are overlapping participles between the phrases of length 2 whose occurrence frequency is less than a preset threshold; If there are overlapping participles between any two phrases of length 2 whose occurrence frequency is less than a preset threshold, determining the position of the overlapping participles in the current log text as a dynamic position; The dynamic position in the current log text is filled with a preset character string to obtain a log template of the current log text.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the log persistent storage method described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for persistent storage of logs as described above is implemented.

[0015] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for persistent storage of logs.

[0016] The present invention provides a log persistent storage method, device, electronic device and storage medium. The method updates a dictionary with a phrase length of a preset length based on the word segmentation of the current log text, and sets the initial value of a variable variable to the preset length; counts the occurrence frequency of each phrase in the dictionary with a phrase length of a variable variable in the current log text, screens the phrases with an occurrence frequency less than a preset threshold, and updates the dictionary with a phrase length of a variable variable based on the phrases with an occurrence frequency less than the preset threshold after reducing the variable variable by 1; repeats the above steps until the value of the variable variable is 1, extracts a log template of the current log text based on a phrase with a phrase length of 2 with an occurrence frequency less than the preset threshold, thereby matching the log template of the current log text with each sample template in a log text library, and after obtaining the sample template matched by the current log text, extracts the variable data of the current log text based on the sample template matched by the current log text, and saves the variable data of the current log text and the identification information of the sample template matched by the current log text in a persistent storage system, thereby reducing the storage space required for the log text and alleviating the storage pressure of the persistent storage of massive logs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1It is a flow chart of the log persistent storage method provided by the present invention; Figure 2 It is a flowchart of the method for determining a preset threshold value provided by the present invention; Figure 3 It is a structural schematic diagram of a log persistent storage device provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 It is a flow chart of the log persistent storage method provided by the present invention, such as Figure 1 As shown, the method includes: Initialization step 110: based on the word segmentation of the current log text, updating the dictionary with a phrase length of a preset length, and setting the initial value of the variable variable to the preset length; Variable data detection step 120: Counting the frequency of occurrence of each phrase in the dictionary whose phrase length is the variable variable in the current log text, filtering the phrase whose occurrence frequency is less than a preset threshold, and updating the dictionary whose phrase length is the variable variable based on the phrase whose occurrence frequency is less than the preset threshold after decrementing the variable variable by 1; Iteration step 130: repeating the variable data detection step until the value of the variable variable is 1; Template extraction step 140: extracting a log template of the current log text based on a phrase with a length of 2 and an appearance frequency less than a preset threshold; Persistent storage step 150: Match the log template of the current log text with each sample template in the log text library. After obtaining the sample template matching the current log text, extract the variable data of the current log text based on the sample template matching the current log text, and save the variable data of the current log text and the identification information of the sample template matching it to the persistent storage system.

[0021] Here, for the current log text, the current log text can be segmented according to spaces, tabs, and line breaks to obtain multiple segmented words of the current log text. Subsequently, the multiple segmented words of the current log text are traversed to update the dictionary with a phrase length of a preset length. Among them, the dictionary with a phrase length of a preset length records phrases with a phrase length of the above preset length (for example, 4) and their frequencies. In the process of traversing the segmented words of the current log text, if a phrase in the dictionary appears, the frequency of the corresponding phrase in the dictionary is increased by 1. In some embodiments, the segmented words of the current log text can be traversed based on a sliding window with a width of a preset length, and the length of each sliding of the sliding window is one segmented word.

[0022] After updating the dictionary with a phrase length of a preset length, a variable variable p can be set, and its initial value can be set to the above preset length. Then, variable data detection is performed on the current log text iteration. Specifically, in each round of iteration, the frequency of occurrence of each phrase in the dictionary with a phrase length of the current value of the variable variable in the current log text can be counted, and then the phrases with an occurrence frequency less than the preset threshold can be screened out, and then the variable variable is subtracted by 1 and the dictionary with a phrase length of the current value of the variable variable is updated based on the phrases with an occurrence frequency less than the preset threshold. Among them, each dictionary records phrases of corresponding lengths and their frequencies. Repeat the above variable data detection steps until the value of the variable variable becomes 1. The specific iterative process of variable data detection can be as follows: while(p>1){ words=Calc&Select(dic p , F); p = p-1; Update(words, dic p ); } Among them, p is a variable variable, dic p is a dictionary with a phrase length of p, F is the current log text, Calc&Select() is used to count the frequency of occurrence of each phrase in the dictionary with a phrase length of p in the current log text and filter out phrases with a frequency less than a preset threshold, and Update() is used to update the dictionary with a phrase length of p (a variable variable minus 1) based on phrases with a frequency less than a preset threshold.

[0023] In some embodiments, the setting of the preset threshold in the variable data detection step is the key to affecting the accuracy of this step. In order to improve the accuracy of the variable data detection step, Figure 2 As shown, the preset threshold can be determined in the following manner: Step 210, obtaining the number of phrases corresponding to each occurrence frequency in the dictionary with a phrase length of 2; Step 220, using the frequency of occurrence as an independent variable and the number of phrases corresponding to each frequency of occurrence as a dependent variable, and determining the first-order derivative of each dependent variable with respect to the corresponding independent variable; Step 230: determine the mutation point of the first-order derivative of each dependent variable with respect to the corresponding independent variable as the preset threshold.

[0024] Specifically, for a dictionary with a phrase length of 2, the number of phrases of each occurrence frequency (1, 2, ..., n) in the dictionary is counted. For example, for an occurrence frequency of 1, the number of phrases with a frequency of 1 in the dictionary is counted, and for an occurrence frequency of 2, the number of phrases with a frequency of 2 in the dictionary is counted... Each occurrence frequency (1, 2, ..., n) is used as an independent variable X, and the number of phrases corresponding to each occurrence frequency is used as a dependent variable Y, and each dependent variable Y is smoothed using a smoothing function (such as a loess function). Subsequently, the first-order derivative dY / dX of each dependent variable relative to the corresponding independent variable is determined. Considering that the frequency of occurrence of phrases in the static part is greater than the frequency of occurrence of phrases in the variable part, the mutation point of the first-order derivative of each dependent variable relative to the corresponding independent variable can be determined as the above-mentioned preset threshold.

[0025] In some embodiments, the first-order derivative of each dependent variable with respect to the corresponding independent variable may be clustered to obtain two clusters, and then the mutation point is determined based on the independent variables corresponding to each first-order derivative in the two clusters. The first-order derivative corresponding to the independent variable less than the mutation point and the first-order derivative corresponding to the independent variable greater than the mutation point belong to different clusters.

[0026] In some other embodiments, clustering of one-dimensional data can be achieved by calculating B[n, 2] based on the following formula:

[0027] Where 1≤i≤n, n is the number of independent variables, 1≤j≤2, B[0, j]=0 and B[i, 0]=0, d i is the first-order derivative of the dependent variable corresponding to the ith independent variable relative to the corresponding independent variable, s(d p ,...,d i ) is the sum of the distances between the first-order derivative of each dependent variable with respect to the corresponding independent variable and the average value of the first-order derivative of each dependent variable with respect to the corresponding independent variable. The p value corresponding to B[n, 2] (i.e., B[p-1, j-1]+s(d p ,...,d i )The p value corresponding to the minimum can be used as the mutation point.

[0028] After the above-mentioned iterative process of variable data detection is completed, and a phrase with a frequency of occurrence less than a preset threshold in the dictionary with a phrase length of 2 is obtained, the log template of the current log text can be extracted based on the phrase with a phrase length of 2 with a frequency of occurrence less than the preset threshold. Among them, the phrase with a phrase length of 2 with a frequency of occurrence less than the preset threshold corresponds to the variable part (i.e., variable) in the current log text, and the other parts in the current log text are static parts, and these static parts constitute the log template of the current log text. In some embodiments, in order to extract the log template of the current log text, it can be determined whether there are overlapping participles between the phrases with a frequency of occurrence less than the preset threshold and a phrase length of 2. If there are overlapping participles between any two phrases with a phrase length of 2 with a frequency of occurrence less than the preset threshold, the position of the overlapping participle in the current log text is determined to be a dynamic position. Subsequently, the dynamic position in the current log text is filled with a preset string, thereby obtaining the log template of the current log text.

[0029] It should be noted that, considering that the log template of the current log text is extracted by obtaining the overlapping word segments between two phrases with a phrase length of 2 whose frequency of occurrence is less than the preset threshold as the dynamic position (i.e., the variable part), in some special cases, for example, if the first word segment or the last word segment of the current log text is a variable part, it will be screened out after the variable data detection process performed by the above iteration, but it will only appear in one phrase, so it cannot be identified by the above template extraction method. In order to overcome this situation and improve the accuracy of log template extraction, when updating the dictionary with a preset length of phrase length based on the word segment of the current log text in the initialization step, the last word segment of the previous log text can be added to the starting position of the current log text, and the first word segment of the next log text can be added to the end position of the current log text, and then the word segment of the current log text is traversed based on the sliding window with a width of the preset length to update the dictionary with a phrase length of the preset length. In this way, if the first or last word segment of the current log text is a variable part, it will be screened out after the variable data detection process performed through the above iteration, and it will appear in at least two phrases, so it can be identified through the above template extraction method.

[0030] After obtaining the log template of the current log text, the log template of the current log text can be matched with each sample template in the log text library to obtain the sample template that matches the current log text. In some embodiments, if the log template of the current log text is completely consistent with any sample template in the log text library, it is determined that the two match; if the matching results between the log template of the current log text and each sample template in the log text library are all mismatched, the log template of the current log text is stored in the log text library as a new sample template and the corresponding identification information is obtained, and the new sample template is the sample template that matches the current log text. Based on the sample template that matches the current log text, the variable data of the current log text (that is, the actual data of the variable part) is extracted, and the variable data of the current log text and the identification information of the sample template that matches it are saved in a persistent storage system.

[0031] In summary, the method provided by the embodiment of the present invention updates a dictionary with a phrase length of a preset length based on the word segmentation of the current log text, and sets the initial value of the variable variable to the preset length; counts the frequency of occurrence of each phrase in the dictionary with a phrase length of a variable variable in the current log text, filters the phrases with an occurrence frequency less than a preset threshold, and updates the dictionary with a phrase length of a variable variable based on the phrases with an occurrence frequency less than the preset threshold after reducing the variable variable by 1; repeats the above steps until the value of the variable variable is 1, extracts the log template of the current log text based on the phrase with a phrase length of 2 with an occurrence frequency less than the preset threshold, thereby matching the log template of the current log text with each sample template in the log text library, and after obtaining the sample template matched by the current log text, extracts the variable data of the current log text based on the sample template matched by the current log text, and saves the variable data of the current log text and the identification information of the sample template matched by the current log text to the persistent storage system, thereby reducing the storage space required for the log text and alleviating the storage pressure of the persistent storage of massive logs.

[0032] A log persistent storage device provided by the present invention is described below. The log persistent storage device described below and the log persistent storage method described above can be referred to each other.

[0033] Based on any of the above embodiments, Figure 3 is a schematic diagram of the structure of the log persistent storage device provided by the present invention, such as Figure 3 As shown, the device comprises: Initialization unit 310, used for updating the dictionary with a phrase length of a preset length based on the word segmentation of the current log text, and setting the initial value of the variable variable to the preset length; The variable data detection unit 320 is used to perform a variable data detection step and an iteration step; wherein the variable data detection step includes counting the frequency of occurrence of each phrase in the dictionary whose phrase length is the variable variable in the current log text, screening the phrase whose occurrence frequency is less than a preset threshold, and subtracting 1 from the variable variable and then updating the dictionary whose phrase length is the variable variable based on the phrase whose occurrence frequency is less than the preset threshold; the iteration step includes repeating the variable data detection step until the value of the variable variable is 1; The template extraction unit 330 is used to extract the log template of the current log text based on the phrases with a length of 2 and an appearance frequency less than a preset threshold; The persistent storage unit 340 is used to match the log template of the current log text with each sample template in the log text library, and after obtaining the sample template matching the current log text, extract the variable data of the current log text based on the sample template matching the current log text, and save the variable data of the current log text and the identification information of the sample template matching it to the persistent storage system.

[0034] The device provided by the embodiment of the present invention updates a dictionary with a phrase length of a preset length based on the word segmentation of the current log text, and sets the initial value of a variable variable to the preset length; counts the frequency of occurrence of each phrase in the dictionary with a phrase length of a variable variable in the current log text, screens the phrases with an occurrence frequency less than a preset threshold, and updates the dictionary with a phrase length of a variable variable based on the phrases with an occurrence frequency less than the preset threshold after reducing the variable variable by 1; repeats the above steps until the value of the variable variable is 1, extracts the log template of the current log text based on the phrase with a phrase length of 2 with an occurrence frequency less than the preset threshold, thereby matching the log template of the current log text with each sample template in the log text library, and after obtaining the sample template matched by the current log text, extracts the variable data of the current log text based on the sample template matched by the current log text, and saves the variable data of the current log text and the identification information of the sample template matched by the current log text to a persistent storage system, thereby reducing the storage space required for the log text and alleviating the storage pressure of the persistent storage of massive logs.

[0035] Based on any of the above embodiments, the step of extracting the log template of the current log text based on the phrase with a length of 2 and an appearance frequency less than a preset threshold comprises: Determine whether there are overlapping participles between the phrases of length 2 whose occurrence frequency is less than a preset threshold; If there are overlapping participles between any two phrases of length 2 whose occurrence frequency is less than a preset threshold, determining the position of the overlapping participles in the current log text as a dynamic position; The dynamic position in the current log text is filled with a preset character string to obtain a log template of the current log text.

[0036] Based on any of the above embodiments, the updating of the dictionary with a phrase length of a preset length based on the word segmentation of the current log text specifically includes: Add the last word segment of the previous log text to the beginning of the current log text, and add the first word segment of the next log text to the end of the current log text; The word segments of the current log text are traversed based on a sliding window with a preset width, and a dictionary with a phrase length of the preset length is updated; wherein the sliding window slides for a length of one word each time.

[0037] Based on any of the above embodiments, the preset threshold is determined based on the following steps: Get the number of phrases corresponding to each frequency of occurrence in the dictionary with a phrase length of 2; Taking the frequency of occurrence as the independent variable and the number of phrases corresponding to each frequency of occurrence as the dependent variable, determine the first-order derivative of each dependent variable with respect to the corresponding independent variable; The mutation point of the first-order derivative of each dependent variable relative to the corresponding independent variable is determined as the preset threshold.

[0038] Based on any of the above embodiments, the mutation point of the first-order derivative of each dependent variable relative to the corresponding independent variable is determined based on the following method: Clustering the first-order derivative of each dependent variable with respect to the corresponding independent variable yields two clusters; The mutation point is determined based on the independent variables corresponding to each first-order derivative in the two clusters; wherein the first-order derivative corresponding to the independent variable smaller than the mutation point and the first-order derivative corresponding to the independent variable greater than the mutation point belong to different clusters.

[0039] Based on any of the above embodiments, matching the log template of the current log text with each sample template in the log text library to obtain the sample template matched by the current log text includes: If the log template of the current log text is completely consistent with any sample template in the log text library, then it is determined that the two match; If the matching results between the log template of the current log text and each sample template in the log text library are all mismatched, the log template of the current log text is stored in the log text library as a new sample template and the corresponding identification information is obtained.

[0040] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a memory 420, a communications interface 430 and a communication bus 440, wherein the processor 410, the memory 420 and the communications interface 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 420 to execute the log persistence storage method, which includes: an initialization step: based on the word segmentation of the current log text, updating the dictionary with a phrase length of a preset length, and setting the initial value of the variable variable to the preset length; a variable data detection step: counting the frequency of occurrence of each phrase in the dictionary with a phrase length of the variable variable in the current log text, screening the phrases with a frequency of occurrence less than a preset threshold, and updating the dictionary with a phrase length of the variable variable based on the phrases with a frequency of occurrence less than the preset threshold after reducing the variable variable by 1; an iteration step: Repeat the variable data detection step until the value of the variable variable is 1; template extraction step: based on a phrase with a phrase length of 2 whose appearance frequency is less than a preset threshold, extract the log template of the current log text; persistent storage step: match the log template of the current log text with each sample template in a log text library, and after obtaining the sample template matched by the current log text, extract the variable data of the current log text based on the sample template matched by the current log text, and save the variable data of the current log text and the identification information of the sample template matched by the current log text to a persistent storage system.

[0041] In addition, the logic instructions in the above-mentioned memory 420 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0042] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the log persistence storage method provided by the above-mentioned methods, and the method includes: an initialization step: based on the word segmentation of the current log text, updating the dictionary with a phrase length of a preset length, and setting the initial value of a variable variable to the preset length; a variable data detection step: counting the frequency of occurrence of each phrase in the dictionary with a phrase length of the variable variable in the current log text, filtering out phrases with an occurrence frequency less than a preset threshold, and subtracting 1 from the variable variable based on The phrase whose occurrence frequency is less than a preset threshold updates the dictionary whose phrase length is the variable variable; iterative step: repeating the variable data detection step until the value of the variable variable is 1; template extraction step: extracting the log template of the current log text based on the phrase whose occurrence frequency is less than the preset threshold and whose phrase length is 2; persistent storage step: matching the log template of the current log text with each sample template in the log text library, after obtaining the sample template matched by the current log text, extracting the variable data of the current log text based on the sample template matched by the current log text, and saving the variable data of the current log text and the identification information of the sample template matched by the current log text to the persistent storage system.

[0043] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the above-mentioned log persistence storage method, the method comprising: an initialization step: based on the word segmentation of the current log text, updating the dictionary with a phrase length of a preset length, and setting the initial value of the variable variable to the preset length; a variable data detection step: counting the frequency of occurrence of each phrase in the dictionary with a phrase length of the variable variable in the current log text, filtering out phrases with a frequency of occurrence less than a preset threshold, and updating the phrase length based on the phrase with a frequency of occurrence less than the preset threshold after decrementing the variable variable by 1. The variable data detection step is a dictionary of the variable variables; an iteration step: repeating the variable data detection step until the value of the variable variable is 1; a template extraction step: extracting the log template of the current log text based on a phrase with a phrase length of 2 and an appearance frequency less than a preset threshold; a persistent storage step: matching the log template of the current log text with each sample template in a log text library to obtain a sample template matched by the current log text, extracting the variable data of the current log text based on the sample template matched by the current log text, and saving the variable data of the current log text and identification information of the sample template matched by the current log text to a persistent storage system.

[0044] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0045] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for persistent storage of logs, characterized in that: include: Initialization step: based on the word segmentation of the current log text, updating the dictionary with a phrase length of a preset length, and setting the initial value of the variable variable to the preset length; Variable data detection step: counting the frequency of occurrence of each phrase in the dictionary whose phrase length is the variable variable in the current log text, screening the phrases whose occurrence frequency is less than a preset threshold, and updating the dictionary whose phrase length is the variable variable based on the phrases whose occurrence frequency is less than the preset threshold after decrementing the variable variable by 1; Iteration step: repeating the variable data detection step until the value of the variable variable is 1; Template extraction step: extracting a log template of the current log text based on a phrase with a length of 2 and an appearance frequency less than a preset threshold; Persistent storage step: match the log template of the current log text with each sample template in the log text library, and after obtaining the sample template matching the current log text, extract the variable data of the current log text based on the sample template matching the current log text, and save the variable data of the current log text and the identification information of the sample template matching it to the persistent storage system.

2. A method for persistent log storage according to claim 1, characterized in that: The extracting of the log template of the current log text based on the phrase with a length of 2 and an appearance frequency less than a preset threshold comprises: Determine whether there are overlapping participles between the phrases of length 2 whose occurrence frequency is less than a preset threshold; If there are overlapping participles between any two phrases of length 2 whose occurrence frequency is less than a preset threshold, determining the position of the overlapping participles in the current log text as a dynamic position; The dynamic position in the current log text is filled with a preset character string to obtain a log template of the current log text.

3. A method for persistent storage of logs according to claim 2, characterized in that: The updating of the dictionary with a phrase length of a preset length based on the word segmentation of the current log text specifically includes: Add the last word segment of the previous log text to the beginning of the current log text, and add the first word segment of the next log text to the end of the current log text; The word segments of the current log text are traversed based on a sliding window with a preset width, and a dictionary with a phrase length of the preset length is updated; wherein the sliding window slides for a length of one word each time.

4. A method for persistent storage of logs according to claim 1, characterized in that: The preset threshold is determined based on the following steps: Get the number of phrases corresponding to each frequency of occurrence in the dictionary with a phrase length of 2; Taking the frequency of occurrence as the independent variable and the number of phrases corresponding to each frequency of occurrence as the dependent variable, determine the first-order derivative of each dependent variable with respect to the corresponding independent variable; The mutation point of the first-order derivative of each dependent variable relative to the corresponding independent variable is determined as the preset threshold.

5. A method for persistent storage of logs according to claim 4, characterized in that: The mutation point of the first-order derivative of each dependent variable relative to the corresponding independent variable is determined based on the following method: Clustering the first-order derivative of each dependent variable with respect to the corresponding independent variable yields two clusters; The mutation point is determined based on the independent variables corresponding to each first-order derivative in the two clusters; wherein the first-order derivative corresponding to the independent variable smaller than the mutation point and the first-order derivative corresponding to the independent variable greater than the mutation point belong to different clusters.

6. A method for persistent storage of logs according to any one of claims 1 to 5, characterized in that: The step of matching the log template of the current log text with each sample template in the log text library to obtain the sample template matched by the current log text includes: If the log template of the current log text is completely consistent with any sample template in the log text library, then it is determined that the two match; If the matching results between the log template of the current log text and each sample template in the log text library are all mismatched, the log template of the current log text is stored in the log text library as a new sample template and the corresponding identification information is obtained.

7. A log persistent storage device, characterized in that: include: An initialization unit, used to update a dictionary with a phrase length of a preset length based on the word segmentation of the current log text, and set an initial value of a variable variable to the preset length; A variable data detection unit, configured to execute a variable data detection step and an iteration step; wherein the variable data detection step includes counting the frequency of occurrence of each phrase in the dictionary whose phrase length is the variable variable in the current log text, screening the phrases whose occurrence frequency is less than a preset threshold, and updating the dictionary whose phrase length is the variable variable based on the phrases whose occurrence frequency is less than the preset threshold after decrementing the variable variable by 1; the iteration step includes repeating the variable data detection step until the value of the variable variable is 1; A template extraction unit, configured to extract a log template of the current log text based on a phrase with a length of 2 and an appearance frequency less than a preset threshold; A persistent storage unit is used to match the log template of the current log text with each sample template in the log text library, and after obtaining the sample template matching the current log text, extract the variable data of the current log text based on the sample template matching the current log text, and save the variable data of the current log text and the identification information of the sample template matching it in the persistent storage system.

8. A log persistent storage device according to claim 7, characterized in that: The extracting of the log template of the current log text based on the phrase with a length of 2 and an appearance frequency less than a preset threshold comprises: Determine whether there are overlapping participles between the phrases of length 2 whose occurrence frequency is less than a preset threshold; If there are overlapping participles between any two phrases of length 2 whose occurrence frequency is less than a preset threshold, determining the position of the overlapping participles in the current log text as a dynamic position; The dynamic position in the current log text is filled with a preset character string to obtain a log template of the current log text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, a log persistence storage method as described in any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a log persistence storage method as described in any one of claims 1 to 6 is implemented.

Images