A real-time log time parsing method for domestic operating systems

Abstract

The present invention relates to the field of time parsing, and specifically provides a method for real-time log time parsing of a domestic operating system, including obtaining multiple time strings, obtaining the log generation time of the source log of the time string as a reference time, marking the standard time of the time string, and merging them into a data set containing all time formats; expanding the data set; building a multi-task learning architecture model based on the ChatGLM‑6B model, and using the data set for training, using the trained model to output the corresponding format string of the time string to be predicted in the log; and using a time conversion tool to convert the format string and the time string into standard time. The method solves the technical problems in the prior art of time parsing in logs that are time-consuming, have a single parsing environment, and have poor real-time performance.

Description

Technical Field

[0001] The invention relates to the field of time analysis, and specifically provides a real-time log time analysis method for a domestic operating system. Background Art

[0002] With the development of new technologies such as cloud computing, microservices, and domestic software and hardware, the architecture of the service side is also changing. The traditional monolithic architecture can no longer meet the needs. The new service architecture requires a complex system architecture, including a combination of a series of components such as service discovery, load balancing, fuse mechanism, current limiting protection, and fault isolation to ensure high availability and low failure rate of services. Therefore, the introduction of automated operation and maintenance tools is bound to become a trend. Monitoring and logging are crucial for service availability and observability. In order to achieve this goal, the enterprise-level monitoring system must have multi-dimensional data collection capabilities and perform data aggregation and analysis from multiple dimensions to obtain sufficient information for locating problems. At the same time, the system also needs to generate real-time log records for events on all business critical paths, store the information on the ground, and retrieve and query it in an efficient way. Therefore, in the monitoring and operation and maintenance system, log parsing is very important, especially in the field of observability, which requires the logs of multiple applications or systems to be parsed, especially for large-scale clusters, where there are many associated systems. When parsing logs, the time of the log content must be parsed first. The traditional way is to develop parsing tools for the log writing time of each system to parse the time in the log and convert it into a unified standard time format for output.

[0003] However, using traditional methods, the configuration of monitoring tools needs to be constantly replaced with parsing templates according to system changes, which is a lot of work and takes up a lot of time for operation and maintenance personnel. In addition, with the development of technologies such as automatic expansion and automatic failover, a large number of new component systems will be generated in the cluster at any time. If traditional methods are used for manual processing, real-time performance cannot be guaranteed and the sustainability requirements of the entire cluster cannot be met. Summary of the invention

[0004] In order to overcome the above defects, the present invention is proposed to solve the technical problems in the prior art that the time parsing method in the log is time-consuming, the parsing environment is single, and the real-time performance is poor.

[0005] The present invention provides a real-time log time parsing method for a domestic operating system, comprising the following steps: S1: obtaining multiple time strings, and obtaining the log generation time of the source log of the time string as a reference time, marking the standard time of the time string, and merging them into a data set containing all time formats; S2: Analyze multiple time components according to the time character string in the data set, define format categories for different expressions of each time component, and obtain a format category definition set containing all time components; S3: According to the format category definition set, all time strings in the data set are labeled with format categories, and the format string corresponding to each time string is generated; S4: Expand the dataset by changing the time component values, replacing separators, and adding noise to all time strings in the dataset, and transforming the standard time, format string, and reference time corresponding to the time string; S5: construct a multi-task learning architecture model based on the ChatGLM-6B model, and use the data set obtained in step S4 to train the multi-task learning architecture model, and perform incremental training and fine-tuning on it; S6: Input the time string to be predicted in the log into the trained multi-task learning architecture model and output the corresponding format string; S7: Use the time conversion tool to convert to standard time based on the format string and time string.

[0006] Further, the step S4 comprises: For all time strings in the data set, change the value of any time component to generate a new time string, and generate a new format string and standard time based on the new time string. Format the "seconds" in the new standard time to "00" to generate the reference time, and expand the data set for the first time. For all format strings in the data set after the first expansion, change the separator between the time components to generate a new time string again, and generate a new format string and standard time based on the new time string, and format the "seconds" in the new standard time as "00" to generate the reference time, and expand the data set for the second time; All format strings in the data set after the second expansion are denoised to generate a new time string again. At the same time, a new format string and standard time are generated based on the new time string, and the "seconds" in the new standard time are formatted as "00" to generate a reference time, and the data set is expanded for the third time.

[0007] Further, the step S5 comprises: S51: Define the main task T1 and auxiliary tasks T2 and T3. The main task T1 is a classification task for time format recognition. The auxiliary task T2 is a sequence labeling task for time component extraction. The auxiliary task T3 is a classification task for time logic extraction, that is, reasoning and inferring the order relationship between two time components. S52: Use the bottom encoder E of the chatGLM-6B model as the shared encoder of the multi-task learning architecture model, and add a specific head network for each task, where the output of the encoder is connected to the input of the three head networks; S53: input the time string in the data set obtained in step S4 into the multi-task learning architecture model, obtain the prediction results of the three head networks respectively, and calculate the loss functions of the three head networks and their combined losses according to the prediction results and the true format string; S54: Calculate the gradient of the loss function with respect to the model parameters, and use the Adam optimization algorithm to update the multi-task learning architecture model parameters, and return to step S53 until the combined loss is less than the preset value and the training is completed.

[0008] further, The head network of the main task T1 includes a convolutional neural network, a bidirectional long short-term memory network, a fusion layer and a classification layer. The shared encoder is connected to the input of the convolutional neural network, the output of the convolutional neural network is connected to the first input of the fusion layer, the output of the shared encoder is connected to the input of the bidirectional long short-term memory network, the output of the bidirectional long short-term memory network is connected to the second input of the fusion layer, the output of the fusion layer is connected to the input of the classification layer, and the classification layer outputs multiple time formats with confidence; The head network of auxiliary task T2 includes a Transformer encoder and a fully connected layer, the input of the Transformer encoder is connected to the output of the shared encoder, and the output of the Transformer encoder is connected to the input of the fully connected layer; The head network of auxiliary task T3 includes a temporal graph, a temporal graph convolutional network encoder, a graph attention layer and a classification layer. The output of the shared encoder is connected to the input of the temporal graph, the output of the temporal graph is connected to the input of the temporal graph convolutional network encoder, the output of the temporal graph convolutional network encoder is connected to the input of the graph attention layer, the output of the graph attention layer is connected to the input of the classification layer, and the classification layer outputs the category of temporal logical reasoning.

[0009] Further, the step S53 includes: Use the cross entropy loss function to calculate the sum of the differences between the prediction results output by the main task T1 head network for each time string in the dataset and the actual format string, as the loss function L1 of the main task T1 head network; Use the conditional random field loss function to calculate the sum of the differences between the prediction results output by the auxiliary task T2 head network for each time string in the dataset and the actual format string as the loss function L2 of the auxiliary task T2 head network; Use the cross entropy loss function to calculate the sum of the differences between the prediction results output by the auxiliary task T3 head network for each time string in the dataset and the actual format string, as the loss function L3 of the auxiliary task T3 head network; Assign weights α, β, γ to each head network and calculate the combined loss L = αL1+βL2+γL3.

[0010] Further, the step S7 comprises: Define the standard time format as yyyy-MM-ddHH:mm:ss; Convert the order of time components in the input time string and format string to the order of time components in the standard time format, and for each time component in the format string, extract the value of the corresponding time component in the time string in turn; If the format category of the time component in the format string is the same as that in the standard time format, directly obtain the corresponding number for filling. If the format category of the time component is different, define and convert it into the corresponding value for filling.

[0011] The working principle and beneficial effects of the present invention are: In the scheme of implementing the present invention, historical data is obtained through steps S1-S4, including a time string data set and a time format data set, the time format data set includes standard time, format string and reference time, and then the obtained historical data is input into the model for training through step S5 to obtain a trained model. When a real-time new log time string is received, through steps S6-S7, the trained model can parse the corresponding format string, and then output the standard time through format conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which: Figure 1 It is a flow chart of the main steps of a domestic operating system real-time log time parsing method of the present invention.

[0013] Figure 2 It is a schematic diagram of a list of format category definition sets in the present invention. DETAILED DESCRIPTION

[0014] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0015] Figure 1 This is a schematic flow chart of the main steps of a method for real-time log time parsing in a domestic operating system of the present invention. Figure 1 As shown, a domestic operating system real-time log time parsing method in this embodiment mainly includes the following steps S1-S7. Historical data is obtained through steps S1-S4, including time string, standard time, format string and reference time, and then the obtained historical data is input into the model for training through step S5 to obtain a trained model. When a real-time new log time string is received, the trained model can parse the corresponding format string through steps S6-S7, and then output the standard time through format conversion.

[0016] S1: Acquire multiple time character strings, and simultaneously acquire the log generation time of the source log of the time character strings as a reference time, mark the standard time of the time character strings, and merge them into a data set containing all time formats.

[0017] A large number of time strings are obtained from historical data from multiple channels, which can contain various time formats. A data set is created based on the time strings and their corresponding standard times, and subsequent annotations and extensions correspond to the data set.

[0018] In one implementation, a time string is obtained from a log of a commonly used application in the monitoring system. For example, "2023-10-2314:30:00", "10 / 23 / 20232:30:00PM" In one implementation, a time string is obtained from the network by crawling, for example, "October 23, 14:30", "20231023T143000Z", etc.

[0019] In this embodiment, each piece of training data includes the log generation time and the time string, which are concatenated as model input to improve the accuracy of model training. The log generation time is used as the reference time of each time string, which can provide a reference range for the model. The time in the log should be within 1 day before and after the reference time. The present invention is aimed at extracting the time format of real-time logs, so this reference value is set to avoid failure or error when parsing time strings such as "12-11-1011:12:12".

[0020] S2: Analyze multiple time components according to the time character string in the data set, define format categories for different representations of each time component, and obtain a format category definition set containing all time components.

[0021] Figure 2 is a schematic diagram of a list of format category definition sets in the present invention, such as Figure 2As shown, the time components in this implementation include six types: year, month, day, hour, minute, and second. Among them, for the "year" time component, the period or era is defined as "gg", the year without the era and without leading zero is defined as "y", the year without the era and with leading zero is defined as "yy", and the four-digit year including the era is defined as "yyyy"; for the "month" time component, the single-digit month without leading zero is defined as "M", the single-digit month with leading zero or the two-digit month is defined as "MM", the abbreviated name of the month in English words is defined as "MMM", and the full name of the month in English words is defined as "MMMM"; for the "day" time component, the single-digit date without leading zero is defined as "d", the single-digit date with leading zero or the two-digit date is defined as "dd", and the abbreviated name of a day of the week in English words is defined as "ddd" , the full name of the English word for a day of the week is defined as "dddd"; for the "hour" time component, the hour with a single digit and no leading zero in the 12-hour system is defined as "h", the hour with a single digit and leading zero in the 12-hour system or the hour with a double digit in the 12-hour system is defined as "hh", the hour with a single digit and no leading zero in the 24-hour system is defined as "H", the hour with a single digit and leading zero in the 24-hour system or the hour with a double digit in the 24-hour system is defined as "HH"; for the "minute" time component, the minute with a single digit and no leading zero is defined as "m", the minute with a single digit and leading zero or the minute with a double digit is defined as "mm"; for the "second" time component, the second with a single digit and no leading zero is defined as "s", the second with a single digit and leading zero or the second with a double digit is defined as "ss", and the second with one decimal place is defined as "f". In actual execution, when the obtained time string is marked as a format string, the f in the time component category is the fractional digit of the second, separated by a decimal point. According to system requirements, the data category can also have a millisecond or microsecond part.

[0022] S3: According to the format category definition set, all time strings in the data set are labeled with format categories, and a format string corresponding to each time string is generated.

[0023] Manually annotate each time string in the dataset or use an annotation tool to clearly indicate the format category corresponding to each time component in each time string. For example, the format string obtained after annotating the time string "2023-04-0514:30:00" is "yyyy-MM-DDHH:mm:ss".

[0024] S4: Expand the dataset by changing the time component values, replacing separators, and adding noise to all time strings in the dataset, and transforming the standard time, format string, and reference time corresponding to the time string. Use data enhancement to expand the dataset to more than 100 million records.

[0025] In one implementation, for all time strings in a data set, the value of any time component is changed to generate a new time string, and a new format string and standard time are generated based on the new time string. The "seconds" in the new standard time are formatted as "00" to generate a reference time, and the data set is expanded for the first time. For each group of strings (including time strings and format strings), 10,000 deduplicated data are randomly generated. For example, for the time string 2023-04-0514:30:00, the time strings expanded in this way include 2023-04-0514:30:01, 2023-04-0514:33:00, etc.

[0026] Furthermore, for all format strings in the first expanded data set, the separators between the time components are changed to generate a new time string again, and a new format string and standard time are generated based on the new time string, and the "seconds" in the new standard time are formatted as "00" to generate a reference time, and the data set is expanded for the second time. For example, for the time string 2023-04-0514:30:00, the time string expanded in this way includes 2023:04:0514:30:00.

[0027] Furthermore, all format strings in the second expanded data set are denoised to generate a new time string again. At the same time, a new format string and standard time are generated based on the new time string, and the "seconds" in the new standard time are formatted as "00" to generate a reference time, and the data set is expanded for the third time. For example, the time string 2023-10-2314:30:00 and its format string yyyy-MM-ddHH:mm:ss are denoised and random characters are added. In this way, the expanded time string 2023-10-23@#14:30:00 corresponds to the format string yyyy-MM-dd@#HH:mm:ss.

[0028] S5: Build a multi-task learning architecture model based on the ChatGLM-6B model, and use the data set obtained in step S4 to train the multi-task learning architecture model, and perform incremental training and fine-tuning on it.

[0029] In one implementation, building a multi-task learning architecture model based on the ChatGLM-6B model includes: S51: define the main task T1 and auxiliary tasks T2 and T3, The main task T1 is a classification task for time format recognition.

[0030] Auxiliary task T2 is a sequence labeling task used to extract temporal components.

[0031] Auxiliary task T3 is a classification task used for temporal logic extraction, that is, reasoning about the chronological relationship between two temporal components.

[0032] S52: Use the underlying encoder E of the chatGLM-6B model as the shared encoder of the multi-task learning architecture model, and add a specific head network for each task. The output of the encoder is connected to the input of the three head networks. The encoder is used to encode the input text sequence x into a shared hidden representation h.

[0033] In one embodiment, the head network of the main task T1 includes a convolutional neural network, a bidirectional long short-term memory network, a fusion layer and a classification layer. The output of the shared encoder is connected to the input of the convolutional neural network, the output of the convolutional neural network is connected to the first input of the fusion layer, the output of the shared encoder is connected to the input of the bidirectional long short-term memory network, the output of the bidirectional long short-term memory network is connected to the second input of the fusion layer, the output of the fusion layer is connected to the input of the classification layer, and the classification layer outputs multiple time formats with confidence.

[0034] In this embodiment, the head network of the main task T1 also adopts a multi-view feature fusion mechanism, which not only uses the hidden representation h output by the shared encoder, but also combines the character-level features and the vocabulary-level features. A multi-channel input classifier C1 is designed, in which each channel specializes in processing one feature.

[0035] Character-level features: Convolutional neural networks (CNNs) are used to extract features at the character level. That is, after the convolutional neural network connects the hidden representation h output by the shared encoder, the numerical value corresponding to each time component in the time string can be mapped to the format category corresponding to the time component.

[0036] Vocabulary-level features: Bi-LSTM is used to capture the sequence information of vocabulary. That is, after the Bi-LSTM is connected to the hidden representation h output by the shared encoder, the sequence information between the time components in the time string can be captured. For example, for the input time string 24-11-24 or 11-24-24, the value corresponding to the "year" time component is 24, and the value corresponding to the "month" time component is 11.

[0037] Fusion layer: The character-level features and word-level features are combined through this fusion layer to obtain a comprehensive feature representation.

[0038] Classification layer: By processing the comprehensive features output by the fusion layer, multiple format categories with confidence are output.

[0039] In one embodiment, the head network of auxiliary task T2 includes a Transformer encoder and a fully connected layer, the input of the Transformer encoder is connected to the output of the shared encoder, and the output of the Transformer encoder is connected to the input of the fully connected layer.

[0040] In this embodiment, the head network of auxiliary task T2 is based on the Transformer sequence tagger, which uses the self-attention mechanism to capture long-distance dependencies in time strings.

[0041] Transformer encoder: takes the feature representation h output by the shared encoder as input and extracts the features required for sequence labeling through multiple self-attention layers and feed-forward networks.

[0042] Annotation layer: A fully connected layer is used to convert the output of the Transformer encoder into a label sequence corresponding to the time component.

[0043] In one embodiment, the head network of auxiliary task T3 includes a timing graph, a timing graph convolutional network encoder, a graph attention layer and a classification layer. The output of the shared encoder is connected to the input of the timing graph, the output of the timing graph is connected to the input of the timing graph convolutional network encoder, the output of the timing graph convolutional network encoder is connected to the input of the graph attention layer, the output of the graph attention layer is connected to the input of the classification layer, and the classification layer outputs the category of temporal logical reasoning.

[0044] Input layer: receives the timing diagram as input.

[0045] Temporal Graph Convolutional Network (T-GCN) Encoder: Combining the advantages of graph convolutional networks (GCN) and recurrent neural networks (RNN), we design the T-GCN encoder, which takes a temporal graph as input and extracts temporal features from the graph. The T-GCN encoder first uses GCN to capture the relationship between entities, and then uses RNN to capture the dynamic changes of the time series.

[0046] Graph attention layer: Attention is adjusted on the output of T-GCN, so that the model can dynamically adjust the attention weight according to the current task, strengthen the key temporal logical relationship, and be able to more accurately reason about the temporal logical relationship between temporal components.

[0047] Entities include time strings and individual time components in format strings. A time string can be derived from multiple format strings with different confidence levels. To convert the correct format, reverse the confidence level.

[0048] Classification layer: Connect the T-GCN encoder, graph attention layer and classification layer in sequence, and finally output the format category of the time component. The operation of the classification layer can be to concatenate the two outputs, and then pass through a series of fully connected layers and non-linear activation functions, and finally map them to the category of temporal logical reasoning.

[0049] In this embodiment, the head network of auxiliary task T3 constructs a relationship graph based on the time components in the time string to represent the basic logical relationship between the time components. On the basis of the relationship graph, the time series information is introduced to construct a timing graph to represent the time series relationship between the time components. Specifically, the following steps are included: Use named entity recognition (NER) technology to extract time components from time strings, define a set of rules or use machine learning methods to extract basic logical relationships between time components, such as "before", "after", etc.; Take the time components as nodes and the relationships between the time components as edges to construct a relationship graph; On the basis of the relationship graph, timestamp information is added to each node (time component) to indicate the occurrence time of the time component; The weights of the edges in the relationship graph are adjusted according to the timestamp information to represent the time series relationship between the time components.

[0050] Through this innovative design, the GNN's ability to encode relational data and T-GCN's ability to process time series data are combined, enabling the model to better understand the logical relationships in time series. The introduction of the graph attention mechanism improves the accuracy and generalization of the model in time logic reasoning tasks. The head network of the auxiliary task T3 will be able to handle time logic reasoning tasks more effectively and improve the performance of the entire multi-task learning architecture. This design not only enhances the model's ability to understand time series data, but also improves the model's accuracy and efficiency in parsing complex time data, bringing significant technical progress to the real-time log time parsing method of domestic operating systems and large model technologies.

[0051] The specific head network of the multi-task learning architecture obtained in step S52 has the following beneficial effects: 1. Improved the accuracy of time format recognition Multimodal learning: By combining the processing information of text in the T1 and T2 head networks and the processing information of images by constructing a time graph in T3, the model can understand and recognize time formats from different perspectives, enhancing the model's ability to understand diverse time representations, especially when processing timestamps in handwriting or images.

[0052] Embedded fusion: The design of the fusion layer enables the model to integrate the processing features of text in the T1 and T2 head networks and the processing features of images by constructing a time series graph in T3, improving the model's generalization ability to recognize different time formats.

[0053] 2. Enhanced the performance of time entity extraction Dynamic attention mechanism: The model can dynamically adjust attention according to the current annotation status, so that it can focus more on key information when extracting temporal entities, reducing the possibility of incorrect annotation.

[0054] Adaptive encoder: The model can adaptively adjust the encoder according to the progress of sequence labeling, improving the ability to understand and process complex time series.

[0055] 3. Improved the accuracy of temporal logic reasoning Temporal Graph Convolutional Network (T-GCN): Combining the advantages of graph convolution and recurrent neural networks, the model can better capture the dynamic relationships and sequential information in time series, and improve the accuracy of temporal logical reasoning.

[0056] Graph Attention Reasoning: Through the graph attention mechanism, the model can more accurately reason about the relationship between temporal entities, which is crucial for understanding and predicting logical relationships in time series.

[0057] 4. Improved the generalization ability of the model By introducing more complex feature extraction and inference mechanisms, the model can better generalize to unseen data and reduce the risk of overfitting.

[0058] 5. Enhanced the model's ability to handle complex time data The innovative network design enables the model to process more complex and diverse temporal data, including data in different formats, from different sources (text, images), and with different temporal relationships.

[0059] 6. Improved system stability and reliability In application scenarios that require precise time recording and management, more accurate time resolution can improve system stability and reliability.

[0060] 7. Optimized resource utilization and computing efficiency By sharing the underlying encoder, the multi-task learning architecture reduces the number of model parameters and optimizes the use of computing resources. At the same time, knowledge sharing between multiple tasks also helps improve training efficiency.

[0061] S53: Input the time character string in the data set obtained in step S4 into the multi-task learning architecture model to obtain the prediction results of the three head networks respectively, and calculate the loss functions of the three head networks and their combined losses according to the prediction results and the true format character string.

[0062] In one implementation, a cross entropy loss function is used to calculate the sum of the differences between the prediction results output by the main task T1 head network for each time string in the data set and the actual format string, as the loss function L1 of the main task T1 head network.

[0063] In one implementation, a CRF (conditional random field) loss function is used to calculate the sum of the differences between the prediction results output by the auxiliary task T2 head network for each time string in the dataset and the actual format string as the loss function L2 of the auxiliary task T2 head network.

[0064] In one implementation, a cross entropy loss function is used to calculate the sum of the differences between the prediction results output by the auxiliary task T3 head network for each time string in the dataset and the actual format string, as the loss function L3 of the auxiliary task T3 head network.

[0065] Assign weights α, β, and γ to each head network and calculate the combined loss L = αL1+βL2+γL3. S54: Calculate the gradient of the loss function with respect to the model parameters, and use the Adam optimization algorithm to update the multi-task learning architecture model parameters, and return to step S53 until the combined loss is less than the preset value and the training is completed.

[0066] S6: Input the time string to be predicted in the log into the trained multi-task learning architecture model and output the corresponding format string.

[0067] In this embodiment, the trained model is used for inference, and the time string to be predicted is input to give a time format string. For example, if the input time string is "23-10-2023 14:30:00", the format string "dd-MM-yyyyHH:mm:ss" is returned.

[0068] S7: Use the time conversion tool to convert to standard time based on the format string and time string.

[0069] In this embodiment, the time is formatted into a standard time format using the format string returned by the model. For example, if the time string is "23-10-2023 14:30:00", the format string obtained is "dd-MM-yyyy HH:mm:ss", and the standard time output by the time conversion tool is "2023-10-23 14:30:00".

[0070] The standard time format is defined as yyyy-MM-ddHH:mm:ss, where the time component types consist of year, month, day, hour, minute, second, and millisecond.

[0071] Convert the order of time components in the input time string and format string to the order of time components in the standard time format. For each time component in the format string, take out the value of the corresponding time component in the time string in turn. If the format category of the time component in the format string is the same as that in the standard time format, the corresponding number is directly obtained for filling. If the format category of the time component is different, it is defined according to the format category and converted into the corresponding number for filling.

[0072] For example, the year can be set to 1970-2069 by default. If the format category of the "year" time component in the obtained format string is "yy", then if the corresponding value in the time string is less than 70, 20 is added to the front, and if it is greater than or equal to 70, 19 is added to convert it to the "yyyy" format category.

[0073] In one implementation, when processing h and hh, it is necessary to first parse whether the end of the format string includes A or P, and then find the corresponding value in the time string.

[0074] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order, and they can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0075] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A real-time log time parsing method for a domestic operating system, characterized in that: The following steps are included: S1: obtaining multiple time strings, and obtaining the log generation time of the source log of the time string as a reference time, marking the standard time of the time string, and merging them into a data set containing all time formats; S2: Analyze multiple time components according to the time character string in the data set, define format categories for different expressions of each time component, and obtain a format category definition set containing all time components; S3: According to the format category definition set, all time strings in the data set are labeled with format categories, and the format string corresponding to each time string is generated; S4: Expand the dataset by changing the time component values, replacing separators, and adding noise to all time strings in the dataset, and transforming the standard time, format string, and reference time corresponding to the time string; S5: construct a multi-task learning architecture model based on the ChatGLM-6B model, and use the data set obtained in step S4 to train the multi-task learning architecture model, and perform incremental training and fine-tuning on it; S6: Input the time string to be predicted in the log into the trained multi-task learning architecture model and output the corresponding format string; S7: Use the time conversion tool to convert to standard time based on the format string and time string.

2. According to the method for real-time log time parsing of a domestic operating system according to claim 1, it is characterized in that: The step S4 comprises: For all time strings in the data set, change the value of any time component to generate a new time string, and generate a new format string and standard time based on the new time string. Format the "seconds" in the new standard time to "00" to generate the reference time, and expand the data set for the first time. For all format strings in the dataset after the first expansion, change the separator between the time components to generate a new time string again, and generate a new format string and standard time based on the new time string. Format the "seconds" in the new standard time as "00" to generate the reference time, and expand the dataset for the second time. All format strings in the data set after the second expansion are denoised to generate new time strings again. At the same time, new format strings and standard time are generated based on the new time strings. The "seconds" in the new standard time are formatted as "00" to generate reference time, and the data set is expanded for the third time.

3. According to the method for real-time log time parsing of a domestic operating system in claim 1, it is characterized in that: The step S5 comprises: S51: Define the main task T1 and auxiliary tasks T2 and T3. The main task T1 is a classification task for time format recognition. The auxiliary task T2 is a sequence labeling task for time component extraction. The auxiliary task T3 is a classification task for time logic extraction, that is, reasoning and inferring the order relationship between two time components. S52: Use the bottom encoder E of the chatGLM-6B model as the shared encoder of the multi-task learning architecture model, and add a specific head network for each task, where the output of the encoder is connected to the input of the three head networks; S53: input the time string in the data set obtained in step S4 into the multi-task learning architecture model, obtain the prediction results of the three head networks respectively, and calculate the loss functions of the three head networks and their combined losses according to the prediction results and the true format string; S54: Calculate the gradient of the loss function with respect to the model parameters, and use the Adam optimization algorithm to update the multi-task learning architecture model parameters, and return to step S53 until the combined loss is less than the preset value and the training is completed.

4. According to the method for real-time log time parsing of a domestic operating system as described in claim 3, it is characterized in that: The head network of the main task T1 includes a convolutional neural network, a bidirectional long short-term memory network, a fusion layer and a classification layer. The shared encoder is connected to the input of the convolutional neural network, the output of the convolutional neural network is connected to the first input of the fusion layer, the output of the shared encoder is connected to the input of the bidirectional long short-term memory network, the output of the bidirectional long short-term memory network is connected to the second input of the fusion layer, the output of the fusion layer is connected to the input of the classification layer, and the classification layer outputs multiple time formats with confidence; The head network of auxiliary task T2 includes a Transformer encoder and a fully connected layer, the input of the Transformer encoder is connected to the output of the shared encoder, and the output of the Transformer encoder is connected to the input of the fully connected layer; The head network of auxiliary task T3 includes a temporal graph, a temporal graph convolutional network encoder, a graph attention layer and a classification layer. The output of the shared encoder is connected to the input of the temporal graph, the output of the temporal graph is connected to the input of the temporal graph convolutional network encoder, the output of the temporal graph convolutional network encoder is connected to the input of the graph attention layer, the output of the graph attention layer is connected to the input of the classification layer, and the classification layer outputs the category of temporal logical reasoning.

5. According to the method for real-time log time parsing of a domestic operating system as described in claim 3, it is characterized in that: The step S53 comprises: Use the cross entropy loss function to calculate the sum of the differences between the prediction results output by the main task T1 head network for each time string in the dataset and the actual format string, as the loss function L1 of the main task T1 head network; Use the conditional random field loss function to calculate the sum of the differences between the prediction results output by the auxiliary task T2 head network for each time string in the dataset and the actual format string as the loss function L2 of the auxiliary task T2 head network; Use the cross entropy loss function to calculate the sum of the differences between the prediction results output by the auxiliary task T3 head network for each time string in the dataset and the actual format string, as the loss function L3 of the auxiliary task T3 head network; Assign weights α, β, γ to each head network and calculate the combined loss L = αL1+βL2+γL3.

6. According to the method for real-time log time parsing of a domestic operating system in claim 1, it is characterized in that: The step S7 comprises: Define the standard time format as yyyy-MM-ddHH:mm:ss; Convert the order of time components in the input time string and format string to the order of time components in the standard time format, and for each time component in the format string, extract the value of the corresponding time component in the time string in turn; If the format category of the time component in the format string is the same as that in the standard time format, directly obtain the corresponding number for filling. If the format category of the time component is different, define and convert it into the corresponding value for filling.

Images