Method, device and storage medium for processing log data of controller in vehicle

Abstract

The application discloses a kind of processing method, device and storage medium of the log data of controller in vehicle.The method comprises the following steps: obtaining the initial log data of controller in vehicle;Determine the target log template matched with initial log data from log template set;Initial log data is filled into target log template, and target log data is obtained, and target log data is stored, wherein, target log data is used for fault diagnosis and adjustment to vehicle.The application solves the technical problem that the efficiency of log data analysis and conversion processing is low.

Description

Technical Field

[0001] This invention relates to the field of vehicles, and more specifically, to a method, apparatus, and storage medium for processing log data of a controller in a vehicle. Background Technology

[0002] As the number of vehicles sold and the number of users continue to rise, the amount of log data from vehicle controllers is enormous. How to transfer and parse this log data has become one of the important directions for major car manufacturers to pursue innovation.

[0003] In related technologies, log collection systems can only perform real-time parsing and transfer of a small amount of log data, failing to support the processing of hundreds of millions of log data per day. This results in long processing times, high resource consumption, and a poor user experience. Furthermore, existing log data parsing methods separate coarse-grained and fine-grained log parsing scenarios, failing to accommodate both. This leads to inaccuracy and high errors in fine-grained log parsing, while increasing resource redundancy in coarse-grained log parsing, resulting in low efficiency in log data parsing and transfer.

[0004] There is currently no effective solution to the problem of low efficiency in parsing and transferring log data. Summary of the Invention

[0005] This invention provides a method, apparatus, and storage medium for processing log data of a controller in a vehicle, to at least solve the technical problem of low efficiency in parsing and transferring log data.

[0006] According to one aspect of the present invention, a method for processing log data of a controller in a vehicle is provided. The method may include: acquiring initial log data of the controller in the vehicle; determining a target log template matching the initial log data from a set of log templates; filling the target log template with the initial log data to obtain target log data; and storing the target log data, wherein the target log data is used for fault diagnosis and adjustment of the vehicle.

[0007] Optionally, determining the target log template matching the initial log data from the log template set includes: splitting the initial log data to obtain at least one sub-log data; identifying at least one log template in the log template set that has the same length as the sub-log data and the same symbol as the starting position of the sub-log data as the target log template set corresponding to the sub-log data; determining the Hamming distance between the sub-log data and the log template, and determining a first log template from the target log template set based on the Hamming distance; matching the sub-log data and the first log template to obtain a first matching result; and determining the target log template based on the first matching result.

[0008] Optionally, determining the target log template based on the first matching result includes: determining the first log template as the target log template in response to the first matching result indicating that the similarity between the sub-log data and the first log template is greater than or equal to a similarity threshold; or determining the target log template based on the length of the common subsequence between the sub-log data and the log template in response to the first matching result indicating that the similarity between the sub-log data and the first log template is less than a similarity threshold.

[0009] Optionally, determining the target log template based on the length of the common subsequence between the sub-log data and the log template includes: determining the length of the common subsequence between the sub-log data and the log template, and determining a second log template based on the length; matching the first log template and the second log template to obtain a second matching result; and determining the target log template based on the second matching result.

[0010] Optionally, determining the target log template based on the second matching result includes: in response to the second matching result indicating that the similarity between the first log template and the second log template is greater than or equal to a similarity threshold, merging the first log template and the second log template to obtain a third log template, and determining the third log template as the target log template.

[0011] Optionally, obtaining the vehicle's initial log data includes: responding to the vehicle's service startup module enabling the query function, querying the vehicle's initial log data through the vehicle's query module, and storing the queried initial log data in the vehicle's queue module; and retrieving the initial log data from the queue module.

[0012] Optionally, the method further includes: in response to the service startup module restarting, the existence of each target log data and the existence of a corresponding target index for each target log data, querying the initial log data starting from the last index of the target index; or in response to the service startup module restarting, the absence of each target log data or the absence of a corresponding target index for each target log data, querying the initial log data starting from a set time.

[0013] Optionally, the method further includes: judging the target index based on the time series to obtain a judgment result; performing a filtering operation on the target index in response to the judgment result indicating that the target index exists; or performing a storage operation on the target index in response to the judgment result indicating that the target index does not exist.

[0014] According to another aspect of the present invention, a processing apparatus for log data of a controller in a vehicle is also provided. The apparatus may include: an acquisition unit for acquiring initial log data of the controller in the vehicle; a determination unit for determining a target log template matching the initial log data from a set of log templates; and a processing unit for filling the target log template with the initial log data to obtain target log data, and storing the target log data, wherein the target log data is used for fault diagnosis and adjustment of the vehicle.

[0015] According to another aspect of the present invention, a vehicle is also provided. This vehicle is used to execute the method for processing log data of a controller in a vehicle according to the embodiments of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform a method for processing log data of a controller in a vehicle according to the embodiments of the present invention.

[0017] In this embodiment of the invention, initial log data of the controller in the vehicle is acquired; a target log template matching the initial log data is determined from a set of log templates; the initial log data is filled into the target log template to obtain target log data, which is then stored. The target log data is used for vehicle fault diagnosis and adjustment. In other words, this embodiment of the invention determines a target log template matching the initial log data from a set of log templates, fills the acquired initial log data into the target log template, obtains parsed target log data, and stores the obtained target log data. This achieves the technical effect of improving the efficiency of log data parsing and transfer processing, and solves the technical problem of low efficiency in log data parsing and transfer processing. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0019] Figure 1 This is a flowchart of a method for processing log data of a controller in a vehicle according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of a method for processing log data of a controller in a vehicle according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of a service startup module according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the structure of a log parsing module according to an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the structure of a log transfer module according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the structure of an exception handling module according to an embodiment of the present invention;

[0025] Figure 7 This is a flowchart illustrating how a query rule module queries log data according to an embodiment of the present invention;

[0026] Figure 8 This is a flowchart of a multi-instance exception handling module querying an index according to an embodiment of the present invention;

[0027] Figure 9 This is a schematic diagram of a device for processing log data of a controller in a vehicle according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] According to an embodiment of the present invention, an embodiment of a method for processing log data of a controller in a vehicle is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] Figure 1 This is a flowchart of a method for processing log data of a controller in a vehicle according to an embodiment of the present invention, such as... Figure 1 As shown, the method may include the following steps:

[0033] Step S102: Obtain the initial log data of the controller in the vehicle.

[0034] In the technical solution provided by step S102 of the present invention, initial log data of the controller in the vehicle can be obtained. The initial log data may include basic vehicle data and vehicle-side logs.

[0035] Optionally, the vehicle's basic data can be the Vehicle Identification Number (VIN), which may include at least information such as the vehicle's manufacturer, model, production year, and equipment. The vehicle-side log can be a record file containing events, faults, operations, and other relevant information recorded by the vehicle's computer system, and may include at least information such as vehicle fault information, operating status, operation records, sensor data, network communication, event logs, and software logs.

[0036] Step S104: Determine the target log template that matches the initial log data from the log template set.

[0037] In the technical solution provided in step S104 of the present invention, a target log template matching the acquired initial log data can be determined from the log template set. The log template set can be a predefined group of log templates used to record specific types of log events. It can include at least common log fields and structures to unify and standardize the format and content of logs, helping developers and system administrators quickly create standard-compliant log entries, making log management and analysis more convenient and efficient. The target log template can be the log template in the log template set that has the highest similarity to the initial log data.

[0038] Optionally, after obtaining the initial log data from the controller in the vehicle, the vehicle's log parsing module determines a target log template that matches the initial log data from the log template set. The log parsing module parses the log data based on coarse-grained Hamming Distance (HAMD) parsing and fine-grained Longest Common Subsequence (LCS) parsing.

[0039] This embodiment can take into account the parsing needs of log data of different granularities. It can reduce parsing time and improve parsing speed for any log data, and avoid wasting resources. Compared with the method of parsing log data in real time and traversing all log templates, the log parsing module can reduce computational complexity while improving parsing speed.

[0040] Step S106: Fill the target log template with the initial log data to obtain the target log data, and store the target log data.

[0041] In the technical solution provided in step S106 of the present invention, initial log data can be filled into the target log template to obtain target log data, and the target log data can be stored. The target log data can be log data parsed from the initial log data, and can be used for vehicle fault diagnosis and adjustment.

[0042] For example, the vehicle's log parsing module can be controlled to parse the acquired initial log data, filling it into the target log template to obtain the target log data. The parsed target log data is then stored in the vehicle's log transfer module. This log transfer module can include a production module, a queue module, and a consumption module. The consumption module encompasses the log parsing module and can employ a multi-consumer approach. Combined with a log template matching algorithm, the parsed log data is ultimately stored in a distributed search and analysis engine (e.g., Elastic Search), facilitating accurate problem localization through later reference to the log data. Furthermore, in addition to using multiple consumers for partitioned log data consumption, a multi-instance deployment approach is employed, allowing multiple services to parse log data simultaneously, ensuring no data loss or duplication, significantly improving the speed of log data processing. It should be noted that no specific restrictions are placed on the type of module that parses the initial log data here.

[0043] In steps S102 to S106 of the present invention, initial log data of the controller in the vehicle is obtained; a target log template matching the initial log data is determined from the log template set; the initial log data is filled into the target log template to obtain target log data, which is then stored. The target log data is used for vehicle fault diagnosis and adjustment. In other words, by determining a target log template matching the initial log data from the log template set, filling the obtained initial log data into the target log template, and storing the obtained target log data, the present invention achieves the technical effect of improving the efficiency of log data parsing and transfer processing, and solves the technical problem of low efficiency in log data parsing and transfer processing.

[0044] The method described in this embodiment will be further described below.

[0045] As an optional embodiment, step S104, determining the target log template matching the initial log data from the log template set, includes: splitting the initial log data to obtain at least one sub-log data; determining at least one log template in the log template set that has the same length as the sub-log data and the same symbol as the starting position of the sub-log data as the target log template set corresponding to the sub-log data; determining the Hamming distance between the sub-log data and the log template, and determining a first log template from the target log template set based on the Hamming distance; matching the sub-log data and the first log template to obtain a first matching result; and determining the target log template based on the first matching result.

[0046] In this embodiment, the acquired initial log data can be split to obtain at least one sub-log data. At least one log template from the log template set that has the same length as the sub-log data and shares the same starting position symbol as the sub-log data can be determined as the target log template set corresponding to the sub-log data. Based on the determined target log template set, the Hamming distance between the sub-log data and the log templates can be determined from the target log template set, and based on the determined Hamming distance, a first log template can be determined. The sub-log data and the first log template can be matched to obtain a first matching result, and based on the first matching result, the target log template can be determined. The sub-log data can be log data obtained by splitting the initial log data using a delimiter. The target log template set can be a group of log templates from the log template set that have the same length as the sub-log data and share the same starting position symbol. The first log template can be the log template from the target log template set that has the largest Hamming distance to the sub-log data.

[0047] Optionally, after obtaining the initial log data, it can be preprocessed based on the user's own set of regular expressions. This involves deleting some original variables from the initial log data and splitting it into word sequences using delimiters (such as Da and Ds). This yields at least one sub-log data, which better facilitates providing clean log data for subsequent log template matching. Preprocessing the log data can improve the quality of log data parsing and reduce the number of log templates.

[0048] For example, after obtaining the sub-log data s, the log template set can be traversed through the prefix tree to obtain the target log template set with the same length as the sub-log data s and the same starting position symbol. Based on the determined target log template set, the first log template t with the highest similarity to the sub-log data s can be determined from the target log template set using a similarity measure (SimDsHAMD(s,t)). The sub-log data and the first log template are then matched to obtain the first matching result. Based on the first matching result, the target log template can be determined.

[0049] As an optional embodiment, determining the target log template based on the first matching result includes: determining the first log template as the target log template in response to the first matching result indicating that the similarity between the sub-log data and the first log template is greater than or equal to a similarity threshold; or determining the target log template based on the length of the common subsequence between the sub-log data and the log template in response to the first matching result indicating that the similarity between the sub-log data and the first log template is less than a similarity threshold.

[0050] In this embodiment, when the first matching result between the sub-log data and the first log template indicates that the similarity between the sub-log data and the first log template is greater than or equal to a similarity threshold, the first log template can be determined as the target log template in response to this condition. When the first matching result between the sub-log data and the first log template indicates that the similarity between the sub-log data and the first log template is less than a similarity threshold, the target log template can be determined based on the length of the common subsequence between the sub-log data and the log template in response to this condition. The similarity threshold can be a user-defined threshold, represented by ε.

[0051] Optionally, when the similarity between the sub-log data and the first log template is greater than or equal to the similarity threshold ε, indicating that the first-level coarse-grained template matching is completely correct and the second-level fine-grained template matching can be skipped, the sub-log data is added to the first log template, and the first log template is determined as the target log template. When the similarity between the sub-log data and the first log template is less than the similarity threshold ε, the second-level fine-grained template matching is performed, and the target log template is determined based on the length of the common subsequence between the sub-log data and the log template.

[0052] As an optional implementation method, determining the target log template based on the length of the common subsequence between the sub-log data and the log template includes: determining the length of the common subsequence between the sub-log data and the log template, and determining a second log template based on the length; matching the first log template and the second log template to obtain a second matching result; and determining the target log template based on the second matching result.

[0053] In this embodiment, when the similarity between the sub-log data and the first log template is less than a similarity threshold, the length of the common subsequence between the sub-log data and the log template can be determined, and based on the determined length, a second log template can be determined. The first log template and the second log template can be matched to obtain a second matching result, and based on the second matching result, a target log template can be determined.

[0054] Optionally, when the similarity between the sub-log data and the first log template is less than the similarity threshold ε, a second-level fine-grained template matching is performed. This second-level matching primarily handles log templates with variable parameter lengths. After obtaining the log templates for comparison based on the inverted index, the second log template can be determined using a similarity metric (SimDaLCS(s,t)). The first and second log templates are then matched to obtain a second matching result. Based on this second matching result, the target log template can be determined.

[0055] As an optional embodiment, determining the target log template based on the second matching result includes: in response to the second matching result indicating that the similarity between the first log template and the second log template is greater than or equal to a similarity threshold, merging the first log template and the second log template to obtain a third log template, and determining the third log template as the target log template.

[0056] In this embodiment, when the second matching result of the first log template and the second log template is used to characterize that the similarity between the first log template and the second log template is greater than or equal to the similarity threshold, in response to the second matching result being used to characterize that the similarity between the first log template and the second log template is greater than or equal to the similarity threshold, the first log template and the second log template can be merged to obtain a third log template, and the third log template is determined as the target log template.

[0057] Optionally, when the similarity between the first log template and the second log template is greater than or equal to the similarity threshold ε, the output nodes of the first log template and the second log template are merged to obtain the third log template. Based on the finally determined third log template, parsing can be performed with reference to the unified third log template, and the parsed log data can be stored in a specified object.

[0058] As an optional embodiment, step S102, obtaining the vehicle's initial log data, includes: responding to the vehicle's service startup module enabling the query function, querying the vehicle's initial log data through the vehicle's query module, and storing the queried initial log data in the vehicle's queue module; and obtaining the initial log data from the queue module.

[0059] In this embodiment, when the vehicle's service startup module enables the query function, in response to the vehicle's service startup module enabling the query function, the vehicle's initial log data can be queried through the vehicle's query module, and the queried initial log data is stored in the vehicle's queue module, and the initial log data can be retrieved from the queue module.

[0060] Optionally, the service startup module in this embodiment can provide three different service triggering methods: timed triggering, startup triggering, and remote call. The startup configuration meets the needs of log data parsing and transfer in different scenarios, and automatic switching between the three modes can be achieved, enabling flexible service startup. Specifically, timed triggering uses a timer for time configuration, startup triggering uses a component (e.g., Application Runner) to automatically trigger the log service, and remote calling uses a mode (e.g., RESTful) for triggering.

[0061] Optionally, the log transfer module in this embodiment includes a production module, which contains three query modules: deep pagination, scrolling query, and shallow pagination. These three query modules are adapted to three different log data volumes and query scenarios, respectively. When the vehicle's service startup module enables the query function, the initial log data of the vehicle can be queried through these three query modules. The three different types of query modules store the queried initial log data in the queue module. The queue module uses a currently popular distributed stream processing platform (e.g., Kafka) queue, and the initial log data can be obtained from the queue module.

[0062] As an optional embodiment, the method further includes: in response to the service startup module restarting, the existence of each target log data and the existence of a corresponding target index for each target log data, querying the initial log data starting from the last index of the target index; or in response to the service startup module restarting, the absence of each target log data or the absence of a corresponding target index for each target log data, querying the initial log data starting from a set time.

[0063] In this embodiment, when the service startup module restarts, each target log data exists, and each target log data has a corresponding target index, in response to the service startup module restarting, each target log data exists, and each target log data has a corresponding target index, the initial log data can be queried starting from the last index of the target index. When the service startup module restarts, each target log data does not exist, or each target log data does not have a corresponding target index, in response to the service startup module restarting, each target log data does not exist, or each target log data does not have a corresponding target index, the initial log data can be queried starting from a set time. The set time can be a pre-set time, such as the earliest time of the day, i.e., midnight. This is only an example and no specific limitation is made to the set time.

[0064] Optionally, the exception handling module in this embodiment may include a queue exception handling module, a query rule module, and a multi-instance exception handling module. To ensure that log data is not duplicated or lost in abnormal scenarios such as service crashes or shutdowns, the queue exception handling module is first controlled to adopt a manual offset commit mode to avoid duplicate and lost log data during the storage queue process. Secondly, the query rule module is controlled to introduce sequential (_seq_no) and time (time) mechanisms to ensure that log data is queried in an orderly manner.

[0065] For example, when the service startup module restarts, it first checks whether the target index and target log data exist. If both the target index and target log data exist, it starts querying from the last record of the target index and the query ends five minutes before the current time. If the target index or target log data does not exist, it starts querying from the earliest time of the day.

[0066] As an optional embodiment, the method further includes: judging the target index based on the time series to obtain a judgment result; performing a filtering operation on the target index in response to the judgment result indicating that the target index exists; or performing a storage operation on the target index in response to the judgment result indicating that the target index does not exist.

[0067] In this embodiment, based on the time series, the target index can be judged to obtain a judgment result. When the judgment result indicates that the target index exists, a filtering operation can be performed on the target index in response to the judgment result indicating that the target index exists. When the judgment result indicates that the target index does not exist, a storage operation can be performed on the target index in response to the judgment result indicating that the target index does not exist.

[0068] Optionally, since the service in this embodiment is deployed on a single machine with multiple instances, the problem of log data duplication caused by multiple instances needs to be considered. It can be based on time series, using time as a unique key for judgment. If the target index exists, it is filtered; if the target index does not exist, it is stored.

[0069] This embodiment first initiates the query of log data in Elastic Search through three triggering methods of the service startup module. Then, it adopts a two-layer framework of coarse-grained and fine-grained template matching to parse the log data. In the log transfer module, a queue and multi-instance architecture are used to ensure fast reading and storage of log data, improving the processing speed of hundreds of millions of log data. Finally, while ensuring the processing speed of log data, the exception handling module adopts three exception mechanisms: queue exception handling module, query rule module, and multi-instance exception handling module, to ensure the reliability of log data, that is, to ensure that log data is not lost or duplicated, thus comprehensively improving the accuracy and speed of log data processing.

[0070] This embodiment acquires initial log data from the controller in the vehicle; determines a target log template matching the initial log data from a set of log templates; fills the target log template with the initial log data to obtain target log data, and stores the target log data. The target log data is used for vehicle fault diagnosis and adjustment. In other words, this embodiment of the invention determines a target log template matching the initial log data from a set of log templates, fills the target log template with the acquired initial log data, obtains parsed target log data, and stores the obtained target log data. This achieves the technical effect of improving the efficiency of log data parsing and transfer processing, and solves the technical problem of low efficiency in log data parsing and transfer processing.

[0071] Example 2

[0072] The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.

[0073] With economic development and the continuous improvement of people's living standards, the number of vehicles sold and the number of car users are constantly increasing. The amount of log data generated by vehicle controllers is enormous, making the need for rapid and accurate log data analysis increasingly apparent in order to accurately locate problems and expose and resolve them early. Therefore, log data transfer and parsing methods have become a key area of ​​innovation for major automakers. Currently, existing log collection systems only perform real-time parsing and transfer of small amounts of log data, unable to support the processing of hundreds of millions of log data per day. This results in long processing times, high resource consumption, and a poor user experience. Furthermore, existing log data parsing methods separate coarse-grained and fine-grained log parsing scenarios, failing to address both simultaneously. This leads to inaccuracy and high errors in fine-grained log parsing, while increasing resource redundancy in coarse-grained log parsing, thus hindering the parsing and transfer of hundreds of millions of log data per day.

[0074] As an alternative example, a log parsing template and a log parsing method based on the template are proposed. The log parsing template corresponds to the Internet Protocol (IP) address of the device. The log parsing template is used to parse logs from different devices into standardized log information with the same format. The log parsing template includes a pre-parsing template and a main parsing template. The pre-parsing template can be either empty or a version parsing sub-template. The version parsing sub-template extracts characters from the log containing the version of the log. The main parsing template corresponding to the extracted characters is selected. The main parsing template is composed of one or more parsing sub-templates. The logs are parsed using the corresponding parsing sub-templates in a specific order to parse the logs into standardized log information with the same format. This method can reduce coding workload, improve work efficiency, and reduce later maintenance costs. However, since this method does not address the parsing requirements of log data with different granularities, it has the problem of not being able to handle the parsing and transfer of hundreds of millions of log data per day.

[0075] As another alternative example, a parallel log parsing method and data synchronization system based on log parsing synchronization are proposed. The source-side data synchronization service includes a log analysis thread and an operation restoration thread, wherein the operation restoration thread executes asynchronously with the log analysis thread. This parallel log parsing method parses the logs of the source database through the log analysis thread to obtain the operation to be synchronized, determines the operation type of the operation to be synchronized, and constructs a shared lock for the operation objects involved in the DML operation when the operation to be synchronized is a Data Manipulation Language (DML) operation, and sends the DML operation and the corresponding shared lock to the operation restoration thread for asynchronous parsing and restoration of the DML operation. When the operation to be synchronized is a Data Definition Language (DDL) operation, an exclusive lock is constructed for the operation objects involved in the DDL operation. After successfully locking the operation objects involved in the DDL operation, the DDL operation is parsed, and the dictionary information of the operation objects pointed to by the DDL operation is maintained and updated. Since this method does not address the parsing requirements of log data of different granularities, it has the problem of not being able to handle the parsing and transfer of hundreds of millions of log data per day.

[0076] As another alternative example, a log parsing method is proposed for use in a distributed computing platform. This method determines the role of each computing node in the platform based on a consensus algorithm (e.g., the Raft protocol), and synchronizes the log parsing rules according to the node roles to ensure that all computing nodes have the same log parsing rules. If log data is received, it is distributed to the computing nodes, which store pre-compiled log parsing rules and control the nodes to perform log parsing operations on the log data. This method can reduce the performance overhead of log parsing operations while ensuring the consistency of log parsing rules. However, since this method does not address the parsing needs of log data with different granularities, it cannot handle the parsing and transfer of hundreds of millions of log data per day.

[0077] To address the aforementioned issues, this embodiment proposes a method for processing log data from a vehicle controller. Given the urgent need to resolve current quality problems and the exponential increase in log data volume, this embodiment proposes a novel method for parsing and transferring hundreds of millions of log data. Furthermore, to address the problem of log data loss and duplication after service downtime or restart due to system anomalies, an effective data processing mechanism is proposed, achieving sub-millisecond accuracy. Considering functional scalability, three service call entry methods are implemented, significantly improving the user experience. This method identifies log templates through streaming and matches them using LCS-based similarity metrics, achieving fast and accurate log data parsing.

[0078] This embodiment, under complex and abnormal backgrounds, achieves the parsing and transfer of hundreds of millions of log data daily through a well-structured framework and lightweight components, improving system throughput. By establishing an effective log data processing mechanism, it enhances system reliability and log data accuracy. It can simultaneously employ three different program triggering entry points: invocation, timed execution, and service startup, meeting the needs of users in various log data processing scenarios. Furthermore, in terms of log data parsing, it uses a streaming approach to find log templates and LCS-based similarity metrics to accurately and quickly match the optimal log template, reducing performance and time overhead associated with log data processing and significantly improving processing speed. Thus, it achieves the technical effect of parsing and transferring hundreds of millions of log data daily, solving the technical problem of being unable to perform parsing and transferring hundreds of millions of log data daily.

[0079] Figure 2 This is a schematic diagram of a method for processing log data of a controller in a vehicle according to an embodiment of the present invention, as shown below. Figure 2As shown in the diagram, the structural schematic of the log data processing method for the controller in this vehicle may include a service startup module 201, a log parsing module 202, a log transfer module 203, and an exception handling module 204. Specifically, after the service startup module 201 starts, the log data is processed by the log parsing module 202, the reading and storage of the log data is handled by the log transfer module 203, and system crashes are handled by the exception handling module 204.

[0080] Figure 3 This is a schematic diagram of the structure of a service startup module according to an embodiment of the present invention, such as... Figure 3 As shown, this embodiment can provide three different service triggering methods: timed triggering, startup triggering, and remote call. Timed triggering uses a timer for time configuration, startup triggering uses the Application Runner component to automatically trigger the log service, and remote calling uses RESTful mode for triggering. By configuring the startup, the needs of log data parsing and transfer in different scenarios can be met, and the three modes can be automatically switched to achieve the purpose of flexible service startup.

[0081] Figure 4 This is a schematic diagram of the structure of a log parsing module according to an embodiment of the present invention, as shown below. Figure 4 As shown, the log parsing module is based on coarse-grained linear HAMD parsing and fine-grained LCS parsing, which can accommodate the parsing needs of log data of different granularities. It can reduce parsing time and improve parsing speed for any log data, avoiding wasted resources. Compared with real-time log data parsing and traversing all log templates, the log parsing module improves parsing speed while reducing computational complexity. This log parsing module provides a two-layer framework log parsing mode. When initial log data is obtained, it can first be preprocessed based on a user-defined set of regular expressions, removing some original variables and splitting the initial log data into word sequences using delimiters Da and Ds, obtaining at least one sub-log data. This better facilitates providing clean log data for subsequent log template matching. Preprocessing is an important step in improving the quality of log data parsing and reducing the number of log templates.

[0082] After obtaining the sub-log data s, the log template set can be traversed through the prefix tree to obtain the target log template set with the same length and starting position as the sub-log data s. Based on the determined target log template set, SimDsHAMD(s,t) can be used to determine the first log template t with the highest similarity to the sub-log data s. Further, the sub-log data and the first log template are matched. When the similarity between the sub-log data and the first log template is greater than or equal to the similarity threshold ε, it indicates that the first layer of coarse-grained template matching is completely correct, and the second layer of fine-grained template matching can be skipped. In this case, the sub-log data is added to the first log template, and the first log template is determined as the target log template.

[0083] When the similarity between the sub-log data and the first log template is less than the similarity threshold ε, a second-level fine-grained template matching is performed. This second-level matching primarily handles log templates with variable parameter lengths. After obtaining the log templates for comparison based on the inverted index, the second log template can be determined using SimDaLCS(s,t). The first and second log templates are then matched to obtain a second matching result. When the similarity between the first and second log templates is greater than or equal to the similarity threshold ε, the output nodes of the first and second log templates are merged to obtain the third log template. Based on the finally determined third log template, parsing can be performed using a unified third log template, and the parsed log data can be stored in a specified object.

[0084] Figure 5 This is a schematic diagram of the structure of a log transfer module according to an embodiment of the present invention, as shown below. Figure 5 As shown in the diagram, the log transfer module's structure includes a production module 501, a queue module 502, and a consumer module 503. The production module 501 contains three query modules: deep pagination, scrolling pagination, and shallow pagination, each suited to different log data volumes and query scenarios. These three different query modules store the initial log data retrieved into the queue module 502, which uses the popular Kafka queue. The consumer module 503 includes a log parsing module. Employing a multi-consumer approach and a log template matching algorithm, it ultimately stores the parsed log data in Elasticsearch, facilitating accurate problem localization through later reference to the log data. Furthermore, in addition to using multiple consumers for partitioned log data consumption, a multi-instance deployment approach is employed, allowing multiple services to parse log data simultaneously while ensuring no data loss or duplication, significantly improving the speed of log data processing.

[0085] Figure 6This is a schematic diagram of the structure of an exception handling module according to an embodiment of the present invention, such as... Figure 6 As shown in the diagram, the exception handling module may include a queue exception handling module 601, a query rule module 602, and a multi-instance exception handling module 603. To ensure that log data is not duplicated or lost in abnormal scenarios such as service crashes or shutdowns, the queue exception handling module 601 first employs a manual offset commit mode to avoid duplication and loss of log data during queue storage. Secondly, the query rule module 602 introduces the _seq_no and time mechanisms to ensure ordered log data retrieval. Finally, in the multi-instance exception handling module 603, since the service in this embodiment is deployed on a single machine with multiple instances, the issue of log data duplication caused by multiple instances needs to be considered.

[0086] Figure 7 This is a flowchart illustrating how a query rule module queries log data according to an embodiment of the present invention, such as... Figure 7 As shown, the flowchart for querying log data using the query rules module may include the following steps:

[0087] Step S701: The service startup module is restarted.

[0088] Step S702: Does the target index exist?

[0089] In step S702 above, when the service startup module restarts, it first determines whether the target index exists. If the target index exists, it proceeds to step S703; otherwise, it proceeds to step S704.

[0090] Step S703: Does the target log data exist?

[0091] In step S703 above, if the target index exists, it is further determined whether the target log data exists. If the target log data exists, proceed to step S705; otherwise, proceed to step S704.

[0092] Step S704: Query from the earliest time of the day.

[0093] In step S704 above, if the target index or target log data does not exist, the query is performed from the earliest time of the day.

[0094] Step S705: Start the query from the last record of the target index.

[0095] In step S705 above, if both the target index and the target log number exist, the query starts from the last record in the target index.

[0096] Step S706: Query the time up to five minutes before the current time.

[0097] In step S706 above, if both the target index and the target log number exist, the query starts from the last record of the target index and ends five minutes before the current time.

[0098] Figure 8 This is a flowchart of a multi-instance exception handling module querying an index according to an embodiment of the present invention, such as... Figure 8 As shown, the process of querying the index by the multi-instance exception handling module may include the following steps:

[0099] Step S801: Query the target index corresponding to the target log data.

[0100] Step S802: Does the target index corresponding to the target log data exist?

[0101] In step S802 above, based on the time series, the judgment is made using time as the unique key. If the target index corresponding to the target log data does not exist, proceed to step S803; otherwise, proceed to step S804.

[0102] Step S803: Store the target index corresponding to the target log data.

[0103] In step S803 above, if the target index corresponding to the target log data does not exist, the target index corresponding to the target log data is stored.

[0104] Step S804: Filter the target index corresponding to the target log data.

[0105] In step S804 above, if the target index corresponding to the target log data exists, the target index corresponding to the target log data is filtered.

[0106] This embodiment first initiates the query of log data in Elastic Search through three triggering methods in the service startup module. Then, it employs a two-layer framework (coarse-grained and fine-grained) to perform template matching, combining log template sets to parse the log data. In the log transfer module, a queue and multi-instance architecture ensure fast reading and storage of log data, improving the processing speed of hundreds of millions of log data. Finally, while ensuring log data processing speed, the exception handling module employs three exception mechanisms—queue exception handling, query rule handling, and multi-instance exception handling—to guarantee the reliability of log data, ensuring no data loss or duplication, thus comprehensively improving the accuracy and speed of log data processing. This embodiment considers the two key indicators of log data processing speed and log data reliability, realizing a complete and reliable log transfer and parsing system, improving log data processing efficiency, ensuring log data integrity, and providing users with crucial log information for troubleshooting.

[0107] This embodiment acquires initial log data from the controller in the vehicle; determines a target log template matching the initial log data from a set of log templates; fills the target log template with the initial log data to obtain target log data, and stores the target log data. The target log data is used for vehicle fault diagnosis and adjustment. In other words, this embodiment of the invention determines a target log template matching the initial log data from a set of log templates, fills the target log template with the acquired initial log data, obtains parsed target log data, and stores the obtained target log data. This achieves the technical effect of improving the efficiency of log data parsing and transfer processing, and solves the technical problem of low efficiency in log data parsing and transfer processing.

[0108] Example 3

[0109] According to an embodiment of the present invention, a device for processing log data of a controller in a vehicle is also provided. It should be noted that this device for processing log data of a controller in a vehicle can be used to execute the method for processing log data of a controller in a vehicle as described in Embodiment 1.

[0110] Figure 9 This is a schematic diagram of a device for processing log data of a controller in a vehicle according to an embodiment of the present invention, as shown below. Figure 9 As shown, the log data processing device 900 of the controller in the vehicle may include: an acquisition unit 902, a determination unit 904, and a processing unit 906.

[0111] Acquisition unit 902 is used to acquire the initial log data of the controller in the vehicle.

[0112] The determination unit 904 is used to determine the target log template that matches the initial log data from the log template set.

[0113] The processing unit 906 is used to fill the initial log data into the target log template to obtain the target log data, and to store the target log data. The target log data is used for vehicle fault diagnosis and adjustment.

[0114] Optionally, the determining unit 904 includes: a first determining module, used to split the initial log data to obtain at least one sub-log data; a second determining module, used to determine at least one log template in the log template set that has the same length as the sub-log data and has the same symbol as the starting position of the sub-log data as the target log template set corresponding to the sub-log data; a third determining module, used to determine the Hamming distance between the sub-log data and the log template, and to determine a first log template from the target log template set based on the Hamming distance; a fourth determining module, used to match the sub-log data and the first log template to obtain a first matching result; and a fifth determining module, used to determine the target log template based on the first matching result.

[0115] Optionally, the fifth determining module includes: a first determining submodule, used to determine the first log template as the target log template in response to the first matching result indicating that the similarity between the sub-log data and the first log template is greater than or equal to a similarity threshold; or a second determining submodule, used to determine the target log template based on the length of the common subsequence between the sub-log data and the log template in response to the first matching result indicating that the similarity between the sub-log data and the first log template is less than a similarity threshold.

[0116] Optionally, the second determining submodule is configured to respond to the first matching result indicating that the similarity between the sub-log data and the first log template is less than a similarity threshold by the following steps: determining the length of the common sub-sequence between the sub-log data and the log template, and determining a second log template based on the length; matching the first log template and the second log template to obtain a second matching result; and determining the target log template based on the second matching result.

[0117] Optionally, the second determining submodule is used to determine the target log template based on the second matching result through the following steps: in response to the second matching result indicating that the similarity between the first log template and the second log template is greater than or equal to a similarity threshold, the first log template and the second log template are merged to obtain a third log template, and the third log template is determined as the target log template.

[0118] Optionally, the acquisition unit 902 includes: a storage module, used to enable the query function in response to the vehicle's service startup module, query the vehicle's initial log data through the vehicle's query module, and store the queried initial log data in the vehicle's queue module; and an acquisition module, used to acquire the initial log data from the queue module.

[0119] Optionally, the device further includes: a first query unit, configured to query the initial log data starting from the last index of the target index in response to a restart of the service startup module, the existence of each target log data and the existence of a corresponding target index for each target log data; or a second query unit, configured to query the initial log data starting from a set time in response to a restart of the service startup module, the non-existence of each target log data or the non-existence of a corresponding target index for each target log data.

[0120] Optionally, the device further includes: a judgment unit, used to judge the target index based on the time series and obtain a judgment result; a filtering unit, used to filter the target index in response to the judgment result indicating that the target index exists; or a storage unit, used to store the target index in response to the judgment result indicating that the target index does not exist.

[0121] In this embodiment of the invention, the initial log data of the controller in the vehicle is acquired by the acquisition unit 902, the determination unit 904 determines the target log template that matches the initial log data from the log template set, and the processing unit 906 fills the initial log data into the target log template to obtain the target log data, which is then stored. The target log data is used for vehicle fault diagnosis and adjustment. In other words, this embodiment of the invention determines the target log template that matches the initial log data from the log template set, fills the acquired initial log data into the target log template, obtains the parsed target log data, and stores the obtained target log data. This achieves the technical effect of improving the efficiency of log data parsing and transfer processing, and solves the technical problem of low efficiency in log data parsing and transfer processing.

[0122] Example 4

[0123] According to an embodiment of the present invention, a vehicle is also provided, which is used to execute the method for processing log data of the controller in any of the vehicles in Embodiment 1.

[0124] Example 5

[0125] According to an embodiment of the present invention, a computer-readable storage medium is also provided, the storage medium including a stored program, wherein the program executes the method for processing log data of the controller in the vehicle in Embodiment 1.

[0126] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0127] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0128] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0129] The units defined as separate components may or may not be physically separate. Similarly, the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0130] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0132] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for processing log data of a controller in a vehicle, characterized in that, include: Obtain the initial log data of the controller in the vehicle; Determine the target log template that matches the initial log data from the log template set; The initial log data is filled into the target log template to obtain the target log data, and the target log data is stored. The target log data is used for fault diagnosis and adjustment of the vehicle. The process of determining a target log template matching the initial log data from a log template set includes: splitting the initial log data to obtain at least one sub-log data; identifying at least one log template in the log template set that has the same length as the sub-log data and the same symbol as the starting position of the sub-log data as the target log template set corresponding to the sub-log data; determining the Hamming distance between the sub-log data and the log template, and determining a first log template from the target log template set based on the Hamming distance; matching the sub-log data and the first log template to obtain a first matching result; and responding to the first matching result being used to characterize the similarity between the sub-log data and the first log template as greater than a certain value. If the similarity is equal to a similarity threshold, the first log template is determined as the target log template; or, in response to the first matching result indicating that the similarity between the sub-log data and the first log template is less than the similarity threshold, the length of the common subsequence between the sub-log data and the log template is determined, and a second log template is determined based on the length; the first log template and the second log template are matched to obtain a second matching result; in response to the second matching result indicating that the similarity between the first log template and the second log template is greater than or equal to the similarity threshold, the first log template and the second log template are merged to obtain a third log template, and the third log template is determined as the target log template.

2. The method according to claim 1, characterized in that, Obtaining the initial log data of the vehicle includes: In response to the service startup module of the vehicle enabling the query function, the initial log data of the vehicle is queried through the query module of the vehicle, and the queried initial log data is stored in the queue module of the vehicle. The initial log data is obtained from the queue module.

3. The method according to claim 1 or 2, characterized in that, The method further includes: When the service startup module restarts, each target log data exists and each target log data has a corresponding target index. The initial log data is queried starting from the last entry in the target index; or When the service startup module restarts, if each of the target log data does not exist or the target index corresponding to each of the target log data does not exist, the initial log data will be queried starting from a set time.

4. The method according to claim 3, characterized in that, The method further includes: Based on the time series, the target index is judged to obtain the judgment result; In response to the determination result indicating the existence of the target index, a filtering operation is performed on the target index; or In response to the determination result indicating that the target index does not exist, a storage operation is performed on the target index.

5. A device for processing log data of a controller in a vehicle, comprising: The acquisition unit is used to acquire the initial log data of the controller in the vehicle; A determining unit is used to determine a target log template that matches the initial log data from the log template set; The processing unit is used to fill the initial log data into the target log template to obtain target log data, and to store the target log data, wherein the target log data is used for fault diagnosis and adjustment of the vehicle; The determining module is further configured to: split the initial log data to obtain at least one sub-log data; determine at least one log template in the log template set that has the same length as the sub-log data and the same symbol as the starting position of the sub-log data as the target log template set corresponding to the sub-log data; determine the Hamming distance between the sub-log data and the log template, and determine a first log template from the target log template set based on the Hamming distance; match the sub-log data and the first log template to obtain a first matching result; and, in response to the first matching result indicating that the similarity between the sub-log data and the first log template is greater than or equal to a similarity threshold, set the... The first log template is determined as the target log template; or, in response to the first matching result indicating that the similarity between the sub-log data and the first log template is less than the similarity threshold, the length of the common subsequence between the sub-log data and the log template is determined, and a second log template is determined based on the length; the first log template and the second log template are matched to obtain a second matching result; in response to the second matching result indicating that the similarity between the first log template and the second log template is greater than or equal to the similarity threshold, the first log template and the second log template are merged to obtain a third log template, and the third log template is determined as the target log template.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes the method for processing log data of the controller in the vehicle according to any one of claims 1 to 4.

Images