Log analysis method and device, electronic equipment and storage medium

Abstract

The invention discloses a log analysis method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the main technical scheme comprises the steps of obtaining log data, and determining a log format of the log data; determining a target log analysis rule according to the log format and the target data item, and analyzing the log data according to the target log analysis rule to obtain the target data item in the log data; and identifying each data item in the log data according to different signal features, and respectively storing the identified data in a database corresponding to each target data item. By configuring the analysis rules of different log formats and different data items in advance and determining the corresponding log analysis rules for log analysis through the log formats and the required data items in practical application, different types of log data are processed through one processing analysis service, the number of analysis services is reduced, and the processing efficiency is improved. Therefore, the manual intervention demand is reduced, and the working efficiency and accuracy are improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a log parsing method and apparatus, electronic device and storage medium. Background Technology

[0002] Log data can come from various sources and systems, including applications, databases, network devices, and operating systems. Each log format may have its unique structure and fields, which requires the parsing service to be able to flexibly handle multiple log types.

[0003] Currently, the process of setting up the parsing service is quite cumbersome, requiring development on the existing basis. Some code will be redeveloped repeatedly, wasting time. If there are many types of log structures to be parsed (there are many parsing services), the configuration, processing and maintenance of each parsing service is also quite troublesome and prone to problems, showing great limitations. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for parsing logs. Its main purpose is to address the problems of cumbersome configuration, processing, and maintenance of log parsing services, which are prone to errors.

[0005] According to a first aspect of this disclosure, a log parsing method is provided, comprising:

[0006] Obtain log data and determine the log format of the log data;

[0007] The target log parsing rules are determined based on the log format and the target data items to be extracted, and the log data is parsed according to the target log parsing rules to obtain the target data items in the log data;

[0008] Based on the signal characteristics of different target data items, each target data item in the log data is identified, and the data corresponding to each identified target data item is stored in the database corresponding to each target data item.

[0009] Optionally, the method further includes:

[0010] Statistical analysis is performed on the historical data in each of the databases to determine whether there are any abnormal events.

[0011] The statistical analysis results corresponding to each of the databases are converted into display data in a preset format and displayed in a preset user interface; wherein, the statistical analysis results include normal information and / or abnormal information.

[0012] Optionally, the step of performing statistical analysis on the historical data in each of the databases to determine whether there is any abnormal information further includes:

[0013] Calculate the average value of historical data in each of the aforementioned databases;

[0014] Based on whether the difference between the data of each target data item and the average data of the corresponding database is greater than or equal to a first preset data threshold, it is determined whether there is abnormal information in the target data item; or

[0015] Whether there is abnormal information is determined by whether the number of times the data of the target data item is greater than or equal to the second preset data threshold is greater than or equal to the preset number threshold.

[0016] Optionally, before determining the target log parsing rule based on the log format and the target data item to be extracted, and parsing the log data according to the target log parsing rule to obtain the target data item in the log data, the method further includes:

[0017] Configure different log parsing rules for different log formats; wherein, each log parsing rule contains at least one regular expression, and different regular expressions are used to parse data for different data items;

[0018] Configure a whitelist of data items to be extracted and a whitelist of vehicles from which log data is to be extracted; wherein, the whitelist of data items contains at least one target data item.

[0019] Optionally, the method further includes:

[0020] If the data items in the whitelist of data items to be extracted change, the log parsing rules are re-determined based on the log format and the target data item, and the log data is parsed again.

[0021] Optionally, before acquiring log data and determining the log format of the log data, the method further includes:

[0022] The log data is preprocessed according to a preset regular expression.

[0023] According to a second aspect of this disclosure, a log parsing apparatus is provided, comprising:

[0024] An acquisition unit is used to acquire log data and determine the log format of the log data;

[0025] The determining unit is used to determine the target log parsing rules based on the log format and the target data items to be extracted.

[0026] The parsing unit is used to parse the log data according to the target log parsing rules to obtain the target data item in the log data;

[0027] The identification unit is used to identify each target data item in the log data according to the signal characteristics of different target data items, and to store the data corresponding to each identified target data item in the database corresponding to each target data item.

[0028] Optionally, the device further includes:

[0029] The statistical analysis unit is used to perform statistical analysis on the historical data in each of the databases to determine whether there is any abnormal information.

[0030] The display unit is used to convert the statistical analysis results corresponding to each of the databases into display data in a preset format and display it in a preset user interface; wherein the statistical analysis results include normal information and / or abnormal information.

[0031] Optionally, the statistical analysis unit is further used for:

[0032] Calculate the average value of historical data in each of the aforementioned databases;

[0033] Based on whether the difference between the data of each target data item and the average data of the corresponding database is greater than or equal to a first preset data threshold, it is determined whether there is abnormal information in the target data item; or

[0034] Whether there is abnormal information is determined by whether the number of times the data of the target data item is greater than or equal to the second preset data threshold is greater than or equal to the preset number threshold.

[0035] Optionally, the device further includes:

[0036] The first configuration unit is configured to configure different log parsing rules for different log formats before the determining unit determines the target log parsing rules based on the log format and the target data item to be extracted; wherein, the log parsing rule contains at least one regular expression, and different regular expressions are used to parse the data of different data items;

[0037] The second configuration unit is used to configure a whitelist of data items to be extracted and a whitelist of vehicles for which log data to be extracted; wherein the whitelist of data items contains at least one target data item.

[0038] Optionally, the device further includes:

[0039] The parsing unit is also used to re-determine the log parsing rules based on the log format and the target data item when the data items in the whitelist of data items to be extracted change, and to re-parse the log data.

[0040] Optionally, before the acquisition unit acquires the log data and determines the log format of the log data, the apparatus further includes:

[0041] The preprocessing unit is used to preprocess the log data according to a preset regular expression.

[0042] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0043] At least one processor; and

[0044] A memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0046] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.

[0047] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0048] The log parsing method, apparatus, electronic device, and storage medium disclosed herein mainly include the following technical solutions: acquiring log data and determining the log format of the log data; determining target log parsing rules based on the log format and the target data items to be extracted, and parsing the log data according to the target log parsing rules to obtain the target data items in the log data; identifying each target data item in the log data based on the signal characteristics of different target data items, and storing the data corresponding to each identified target data item in the database corresponding to each target data item. Compared with related technologies, the embodiments of this application, by pre-configuring parsing rules for different log formats and different data items, and determining the corresponding log parsing rules for log parsing based on the log format and the required data items in actual application, realizes the processing of different types of log data through a single parsing service, reduces the number of parsing services, thereby reducing the need for manual intervention and improving work efficiency and accuracy.

[0049] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0050] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0051] Figure 1 A flowchart illustrating a log parsing method provided in an embodiment of this disclosure;

[0052] Figure 2 A flowchart illustrating another log parsing method provided in an embodiment of this disclosure;

[0053] Figure 3 A flowchart illustrating another log parsing method provided in an embodiment of this disclosure;

[0054] Figure 4 A schematic diagram of the structure of a log parsing device provided in an embodiment of this disclosure;

[0055] Figure 5 A schematic diagram of another log parsing apparatus provided in an embodiment of this disclosure;

[0056] Figure 6 A schematic block diagram of an example electronic device provided for embodiments of this disclosure. Detailed Implementation

[0057] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0058] The following description, with reference to the accompanying drawings, outlines a log parsing method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure.

[0059] Figure 1 This is a flowchart illustrating a log parsing method provided in an embodiment of the present disclosure.

[0060] like Figure 1 As shown, the method includes the following steps:

[0061] Step 101: Obtain log data and determine the log format of the log data.

[0062] In some embodiments, the log data may be log data of an application, server, or device, and may be obtained through database queries or API (Application Programming Interface) interfaces. Specifically, this application embodiment does not limit this.

[0063] In some embodiments, log data typically includes fields such as timestamp, log level, and message content, and may have different log formats, such as common text log formats, JSON (JavaScript Object Notation) formats, XML (Extensible Markup Language) formats, etc.; specifically, this application embodiment does not limit the specific format of log data.

[0064] Step 102: Determine the target log parsing rule based on the log format and the target data item to be extracted, and parse the log data according to the target log parsing rule to obtain the target data item in the log data.

[0065] Parsing log data: Extract key information from the preprocessed data according to the rules of log parsing, such as signal generation time, log level, and corresponding service. These can be extracted using relatively simple parsing rules, while signal name and value require more complex parsing rules.

[0066] Parsing configuration: Configure multiple regular expressions required for parsing based on the log format.

[0067] For example: Extracting the main content from logs based on log format characteristics: Here are two examples:

[0068] 1.^CMLog:(? <content>The +)$ option extracts the content following CMLog: and adds the detailed content to the content field.

[0069] 2.\s*(? <signalname>\S+?)=\s*(? <signalvalue>[-0-9a-zA-Z.]*),?

[0070] This regular expression means that its purpose is to match key-value pairs where the key and value are connected by an equals sign, leading and trailing whitespace is allowed, and a comma is permitted at the end. Within a text string, this regular expression can be used to extract keys and values.

[0071] Suppose we have the following log lines:

[0072] 2024-03-20 08:30:15INFO:Application started successfully

[0073] Based on the above log format, the following target log parsing rules can be defined:

[0074] 1. Timestamp: Extracts the date and time portion from a log line.

[0075] 2. Log Level: Extract the log level (INFO) portion from the log line.

[0076] 3. Message content: Extract the message content portion after the colon from the log line.

[0077] Based on these rules, the above log lines can be parsed to obtain the target data items:

[0078] -Timestamp: 2024-03-20 08:30:15

[0079] - Log level: INFO

[0080] -Message content: Application started successfully

[0081] By defining target log parsing rules and applying these rules to parse log data, the desired target data items can be effectively extracted, enabling further analysis and processing of the log data.

[0082] Step 103: Identify each target data item in the log data according to the signal characteristics of different target data items, and store the data corresponding to each identified target data item in the database corresponding to each target data item.

[0083] Because a parsing service involves multiple parsing requirements, the signals corresponding to each requirement must be stored separately during storage. The parsed data will be identified according to the signal characteristics of each requirement. At the same time, the relevant database configurations must be matched one by one, and finally the signals will be stored in the corresponding databases and data tables to make the signal storage clearer and improve the efficiency of subsequent data processing.

[0084] For example: the signal characteristic is reduced torque, and the reduced torque signal characteristic is a signal starting with Vb, such as: VbDTA_ReAxleDecCtrlSt_flg.

[0085] Motor: Divided into power-back and power-out. If the VbHPR_HVDchrgPredRgnEna_flg signal value is 1, it indicates that the signal value corresponding to this log line is a power-back related signal. If the VbHPR_HVDchrgPredDrvEna_flg signal value is 1, it indicates that the signal value corresponding to this log line is a power-out related signal.

[0086] Once each target data item is identified and extracted, it can be stored in its corresponding database. For example, timestamps can be stored in a time-series database, log levels can be stored in a relational database, and message content can be stored in a text database or search engine for subsequent querying and analysis. Specifically, this application does not limit the form of the database.

[0087] The log parsing method disclosed herein mainly includes the following technical solutions: acquiring log data and determining the log format of the log data; determining target log parsing rules based on the log format and the target data items to be extracted, and parsing the log data according to the target log parsing rules to obtain the target data items in the log data; identifying each target data item in the log data based on the signal characteristics of different target data items, and storing the data corresponding to each identified target data item in the database corresponding to each target data item. Compared with related technologies, the embodiments of this application, by pre-configuring parsing rules for different log formats and different data items, and determining the corresponding log parsing rules for log parsing based on the log format and the required data items in actual application, realizes the processing of different types of log data through a single parsing service, reduces the number of parsing services, thereby reducing the need for manual intervention and improving work efficiency and accuracy.

[0088] In some embodiments, after obtaining log data, analysis can be performed on the log data to determine if any anomalies exist. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 Optionally, as shown in the flowchart of the log parsing method provided in this application embodiment, the method further includes:

[0089] Step 101: Obtain log data and determine the log format of the log data.

[0090] Step 102: Determine the target log parsing rule based on the log format and the target data item to be extracted, and parse the log data according to the target log parsing rule to obtain the target data item in the log data.

[0091] Step 103: Identify each target data item in the log data according to the signal characteristics of different target data items, and store the data corresponding to each identified target data item in the database corresponding to each target data item.

[0092] For detailed implementation processes of steps 101-103, please refer to the above description. The embodiments of this application will not be described in detail here.

[0093] Step 104: Perform statistical analysis on the historical data in each of the databases to determine whether there is any abnormal information.

[0094] After log data is stored in the database, it can be compared with historical data using a probabilistic statistical model. The historical data can be used as a sample for statistical analysis, and the overall data distribution can be observed to detect abnormal events. Alerts for abnormal data can be issued as needed, or monitoring can be performed in the form of charts.

[0095] In some embodiments, determining whether anomalous data exists may be performed according to the following steps:

[0096] Calculate the average value of historical data in each of the aforementioned databases;

[0097] Based on whether the difference between the data of each target data item and the average data of the corresponding database is greater than or equal to a first preset data threshold, it is determined whether there is abnormal information in the target data item; or

[0098] Whether there is abnormal information is determined by whether the number of times the data of the target data item is greater than or equal to the second preset data threshold is greater than or equal to the preset number threshold.

[0099] In some embodiments, the amount of data can be statistically analyzed over a day or several days to observe whether the amount of data tends to stabilize. When a particularly high or low amount of data is determined, a first preset data threshold can be set to determine whether the difference between the amount of data and the average amount of data is too large. If the difference is greater than or equal to the first preset data threshold, it is determined that the data is abnormal.

[0100] In some embodiments, certain specific signals can be statistically analyzed to count the number of times the data of a specific signal exceeds a second preset data threshold, or to count the number of times a certain signal value equals a certain number. These indicators can all be used as alarm standards.

[0101] It should be noted that the first preset data threshold and the second preset data threshold are empirical values, which can be set according to actual needs in practical applications. This application embodiment does not limit this.

[0102] In some embodiments, anomaly detection and visualization are mainly focused on log data, but the log parsing process also needs to be monitored. For example, performance monitoring can be performed using Prometheus, as well as monitoring Kafka latency, the number of files parsed, file parsing time, result storage time, and the frequency of certain anomalies (such as incomplete log lines). When monitoring metrics show anomalies, problems can be investigated and dealt with in a timely manner.

[0103] Step 105: Convert the statistical analysis results corresponding to each database into display data in a preset format and display it in a preset user interface; wherein, the statistical analysis results include normal information and / or abnormal information.

[0104] Data visualization can quickly uncover important information from large amounts of data. Presenting data in the form of charts and graphs makes trends and anomalies more intuitive; it also makes it easier to compare differences and similarities between data points. For example, bar charts and pie charts can be used to compare the frequency of signal occurrences, while line charts and trend charts can be used to compare signal trends. It also allows for a better understanding of the relationships and patterns between data. For instance, scatter plots can more intuitively show the relationship between two variables, and heatmaps can more intuitively show the density and distribution of data; however, the specific implementation of this application does not limit this.

[0105] In some embodiments, during log printing, a single log line may contain many signals. In this case, it is necessary to configure a whitelist of signals to be extracted to avoid storing unnecessary data. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 A flowchart illustrating a log parsing method provided in this application embodiment includes:

[0106] Step 106: Configure different log parsing rules for different log formats; wherein, the log parsing rule contains at least one regular expression, and different regular expressions are used to parse data for different data items.

[0107] In some embodiments, specific parsing rules are designed in advance for each log format, including using different regular expressions to extract data for different data items. For each log format, parsing rules containing regular expressions can be designed, with each regular expression used to extract data for a specific data item. For example, for one log format, regular expressions can be used to parse timestamp data, while for another log format, different regular expressions are used to parse log-level data.

[0108] By configuring different log parsing rules for different log formats, the system can more accurately extract and parse various data items, thereby achieving efficient processing and analysis of diverse log data. This flexible configuration method improves the system's applicability and scalability, enabling it to adapt to different types and structures of log data.

[0109] Step 107: Configure the whitelist of data items to be extracted and the whitelist of vehicles from which log data to be extracted; wherein the whitelist of data items contains at least one target data item.

[0110] During log printing, a single log line may contain numerous signals. Therefore, it's necessary to configure a whitelist of signals to be extracted to avoid storing unnecessary data. For some needs involving statistical analysis of vehicle data characteristics, extracting data from all vehicles would result in a massive data volume and consume excessive machine resources. In such cases, a whitelist of vehicles needs to be added (the number can range from several thousand to tens of thousands of relatively active vehicles, determined based on specific requirements). After the initial filtering is completed, signals will be extracted and stored according to the configured parsing rules, preparing for subsequent anomaly analysis and visualization.

[0111] Step 101: Obtain log data and determine the log format of the log data.

[0112] Step 102: Determine the target log parsing rule based on the log format and the target data item to be extracted, and parse the log data according to the target log parsing rule to obtain the target data item in the log data.

[0113] Step 103: Identify each target data item in the log data according to the signal characteristics of different target data items, and store the data corresponding to each identified target data item in the database corresponding to each target data item.

[0114] For detailed implementation processes of steps 101-103, please refer to the above description. The embodiments of this application will not be described in detail here.

[0115] Step 104: Perform statistical analysis on the historical data in each of the databases to determine whether there is any abnormal information.

[0116] Step 105: Convert the statistical analysis results corresponding to each database into display data in a preset format and display it in a preset user interface; wherein, the statistical analysis results include normal information and / or abnormal information.

[0117] For details on the implementation of steps 101-105, please refer to the detailed description in the above application embodiments. This application embodiment will not repeat them here.

[0118] In some embodiments, the method further includes:

[0119] If the data items in the whitelist of data items to be extracted change, the log parsing rules are re-determined based on the log format and the target data item, and the log data is parsed again.

[0120] In some embodiments, before parsing the log data, it is necessary to determine whether the log content is complete and whether the log format is correct; otherwise, the analysis results of subsequent data may be inaccurate. During cleaning, data cleaning can be performed according to the following steps: preprocessing the log data according to a preset regular expression.

[0121] When extending the system to preprocess log data according to preset regular expressions, some preprocessing operations can be performed on the raw log data before data parsing in order to better adapt to and match the preset regular expressions.

[0122] In some embodiments, data preprocessing is performed in two ways. First, data is processed at the data source end, where it is cleaned and standardized at the log collection end. Before writing each line of log data, the timestamp and format are checked for compliance; if they are, the data is written normally; otherwise, it is not. Second, data is cleaned in the cloud. After the logs are uploaded to the cloud, before parsing, regular expressions are used to check the timestamp and format of each line of log data for compliance. For example:

[0123] Timestamp (down to milliseconds) Log level Service name Log content (seconds milliseconds multiple signal names signal values)

[0124] Then decide whether to store it.

[0125] The main purpose of this step is to ensure data quality, avoid storing unnecessary dirty data, and save database storage space. These data preprocessing steps make the raw log data more standardized and cleaner, which is beneficial for subsequent regular expression matching and data parsing. This improves the efficiency and accuracy of data processing, ensuring that the system can accurately extract the required data items for subsequent analysis and application.

[0126] Corresponding to the log parsing method described above, this invention also proposes a log parsing apparatus. Since the apparatus embodiments of this invention correspond to the method embodiments described above, details not disclosed in the apparatus embodiments can be referred to in the method embodiments described above, and will not be repeated here.

[0127] Figure 4 This is a schematic diagram of the structure of a log parsing device provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, it includes:

[0128] Acquisition unit 21 is used to acquire log data and determine the log format of the log data;

[0129] Determining unit 22 is used to determine the target log parsing rules based on the log format and the target data items to be extracted;

[0130] Parsing unit 23 is used to parse the log data according to the target log parsing rules to obtain the target data item in the log data;

[0131] The identification unit 24 is used to identify each target data item in the log data according to the signal characteristics of different target data items, and to store the data corresponding to each identified target data item in the database corresponding to each target data item.

[0132] The log parsing apparatus disclosed herein mainly includes the following technical solutions: acquiring log data and determining the log format of the log data; determining target log parsing rules based on the log format and the target data items to be extracted, and parsing the log data according to the target log parsing rules to obtain the target data items in the log data; identifying each target data item in the log data based on the signal characteristics of different target data items, and storing the data corresponding to each identified target data item in the database corresponding to each target data item. Compared with related technologies, the embodiments of this application, by pre-configuring parsing rules for different log formats and different data items, and determining the corresponding log parsing rules for log parsing based on the log format and the required data items in actual application, realizes the processing of different types of log data through a single parsing service, reduces the number of parsing services, thereby reducing the need for manual intervention and improving work efficiency and accuracy.

[0133] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5 As shown, the device further includes:

[0134] Statistical analysis unit 25 is used to perform statistical analysis on the historical data in each of the databases to determine whether there is any abnormal information;

[0135] The display unit 26 is used to convert the statistical analysis results corresponding to each of the databases into display data in a preset format and display it in a preset user interface; wherein the statistical analysis results include normal information and / or abnormal information.

[0136] Furthermore, in one possible implementation of this disclosure embodiment, the statistical analysis unit 25 is further configured to:

[0137] Calculate the average value of historical data in each of the aforementioned databases;

[0138] Based on whether the difference between the data of each target data item and the average data of the corresponding database is greater than or equal to a first preset data threshold, it is determined whether there is abnormal information in the target data item; or

[0139] Whether there is abnormal information is determined by whether the number of times the data of the target data item is greater than or equal to the second preset data threshold is greater than or equal to the preset number threshold.

[0140] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5 As shown, the device further includes:

[0141] The first configuration unit 27 is configured to configure different log parsing rules for different log formats before the determining unit 22 determines the target log parsing rules based on the log format and the target data item to be extracted; wherein, the log parsing rule contains at least one regular expression, and different regular expressions are used to parse the data of different data items;

[0142] The second configuration unit 28 is used to configure a whitelist of data items to be extracted and a whitelist of vehicles for which log data to be extracted; wherein the whitelist of data items contains at least one target data item.

[0143] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5 As shown, the device further includes:

[0144] The parsing unit 29 is also used to re-determine the log parsing rules according to the log format and the target data item when the data items in the whitelist of data items to be extracted change, and to re-parse the log data.

[0145] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5 As shown, before the acquisition unit 21 acquires log data and determines the log format of the log data, the device further includes:

[0146] The preprocessing unit 210 is used to preprocess the log data according to a preset regular expression.

[0147] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of this disclosure, and the principle is the same. Therefore, the embodiments of this disclosure are not limited thereto.

[0148] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0149] Figure 6 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] like Figure 6 As shown, device 300 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 302 or a computer program loaded from storage unit 308 into RAM (Random Access Memory) 303. RAM 303 can also store various programs and data required for the operation of device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O (Input / Output) interface 305 is also connected to bus 304.

[0151] Multiple components in device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of monitors, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as log parsing methods. For example, in some embodiments, the log parsing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the aforementioned log parsing method by any other suitable means (e.g., by means of firmware).

[0153] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0157] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0158] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0159] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0160] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.< / signalvalue> < / signalname> < / content>

Claims

1. A log parsing method, characterized in that, include: Obtain log data and determine the log format of the log data; The target log parsing rules are determined based on the log format and the target data items to be extracted, and the log data is parsed according to the target log parsing rules to obtain the target data items in the log data; Based on the signal characteristics of different target data items, each target data item in the log data is identified, and the data corresponding to each identified target data item is stored in the database corresponding to each target data item.

2. The method according to claim 1, characterized in that, After storing the data corresponding to each identified target data item in the database corresponding to each target data item, the method further includes: Statistical analysis was performed on the historical data in each of the aforementioned databases to determine whether any abnormal information existed. The statistical analysis results corresponding to each of the databases are converted into display data in a preset format and displayed in a preset user interface; wherein, the statistical analysis results include normal information and / or abnormal information.

3. The method according to claim 2, characterized in that, The step of performing statistical analysis on the historical data in each of the databases to determine whether there is any abnormal information also includes: Calculate the average value of historical data in each of the aforementioned databases; Based on whether the difference between the data of each target data item and the average data of the corresponding database is greater than or equal to a first preset data threshold, it is determined whether there is abnormal information in the target data item; or Whether there is abnormal information is determined by whether the number of times the data of the target data item is greater than or equal to the second preset data threshold is greater than or equal to the preset number threshold.

4. The method according to claim 1, characterized in that, Before determining the target log parsing rule based on the log format and the target data item to be extracted, and parsing the log data according to the target log parsing rule to obtain the target data item in the log data, the method further includes: Configure different log parsing rules for different log formats; wherein, each log parsing rule contains at least one regular expression, and different regular expressions are used to parse data for different data items; Configure a whitelist of data items to be extracted and a whitelist of vehicles from which log data is to be extracted; wherein, the whitelist of data items contains at least one target data item.

5. The method according to claim 4, characterized in that, After acquiring log data and determining the log format of the log data, the method further includes... If the data items in the whitelist of data items to be extracted change, the log parsing rules are re-determined based on the log format and the target data item, and the log data is parsed again.

6. The method according to claim 1, characterized in that, Before acquiring log data and determining the log format of the log data, the method further includes: The log data is preprocessed according to a preset regular expression.

7. A log parsing apparatus, characterized in that, include: An acquisition unit is used to acquire log data and determine the log format of the log data; The determining unit is used to determine the target log parsing rules based on the log format and the target data items to be extracted. The parsing unit is used to parse the log data according to the target log parsing rules to obtain the target data item in the log data; The identification unit is used to identify each target data item in the log data according to the signal characteristics of different target data items, and to store the data corresponding to each identified target data item in the database corresponding to each target data item.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images