Structured analysis and semantic template normalization method for unmanned vehicle operation logs

Abstract

The invention provides a structured analysis and semantic template normalization method for an unmanned vehicle operation log. For the problems of unstructured unmanned vehicle log data, changeable formats, complex semantics and the like, logs are converted into semantic vectors by adopting a multi-language sentence vector coding model, and semantic grouping is performed by using a small-batch K-means clustering algorithm after dimension reduction is performed by adopting an incremental principal component analysis method; extracting a variable field from a clustering result by using a regular expression and generating a standardized template; and intelligent template combination is realized by calculating cosine similarity and editing distance between the templates. The method has the following three technical characteristics: 1) the template consistency is improved by adopting a strategy of clustering first and then normalizing; 2) introducing double similarity constraints to reduce template redundancy; and 3) retaining variable sequence and semantic information to support subsequent analysis. The method is suitable for log data analysis and intelligent operation and maintenance of various complex systems.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing in intelligent transportation systems, and specifically relates to a structured parsing and semantic template normalization method for unmanned vehicle operation logs. Background Art

[0002] With the rapid development of autonomous driving technology and intelligent logistics systems, unmanned vehicles have been widely used in scenarios such as industrial parks, warehousing and transportation, and smart factories. During actual operation, unmanned vehicles continuously generate large amounts of log data, which records task scheduling, path planning, status feedback, module calls, and exception information. These logs not only provide a true reflection of system behavior but also serve as an important foundation for task backtracking, exception diagnosis, and performance optimization. In-depth analysis of log data helps identify system bottlenecks, monitor operational status, and improve system stability and efficiency.

[0003] Existing log parsing methods fall into three main categories: rule-driven parsing, template extraction, and machine learning methods based on semantic modeling. Rule-driven approaches (such as regular expression-based matching) are suitable for log environments with fixed structures, but they suffer from high manual maintenance costs, poor adaptability, and difficulty processing complex logs with frequently changing formats. Template extraction methods such as Drain and Spell utilize fixed-depth tree structures or the longest common subsequence algorithm to extract template structures and annotate variable positions. These methods are suitable for general system logs, but when faced with the complex semantics and nested structures of autonomous vehicle logs, they often suffer from template mis-merging, incomplete variable extraction, and semantic loss, resulting in template segmentation accuracy of only 65%. Some research has attempted to introduce deep learning models such as BERT and LSTM for log semantic modeling. While these have achieved some success in template classification and anomaly detection, the model training costs are high, and processing 100,000 logs takes an hour, making them inadequate for real-time processing.

[0004] Compared to traditional information systems, the log records in autonomous vehicle systems have stronger temporal correlations and contextual dependencies. A single log entry often cannot independently reflect business behavior and requires semantic restoration through the integration of adjacent log entries. Furthermore, the log content is embedded with a large amount of parameter information related to scheduling strategies, perception results, status feedback, and other factors. These complex and inconsistent log entries are difficult to abstract and summarize using a common template. Furthermore, in actual operation, log formats can change frequently depending on system versions, task types, and module call paths, further complicating parsing.

[0005] Unmanned vehicle systems face logs with diverse sources, complex semantics, and diverse formats, placing higher demands on log parsing capabilities. Existing methods often fail to accurately capture key semantic structures, making it difficult to achieve effective template normalization and variable reconstruction, which in turn impacts the reliability and effectiveness of subsequent tasks such as process modeling, anomaly detection, and system visualization. This, to a certain extent, hinders the in-depth application of unmanned vehicle systems in intelligent operations and maintenance, as well as task scheduling optimization.

[0006] To meet the above needs, a unmanned vehicle operation log parsing method with semantic understanding ability, structural abstraction ability and high adaptability is needed. It can achieve template-level semantic normalization and accurate variable extraction while maintaining processing efficiency, and provide reliable data support for subsequent log analysis tasks. Summary of the Invention

[0007] This paper aims to provide a structured parsing and semantic template normalization method for autonomous vehicle operation logs, aiming to address the unstructured nature of autonomous vehicle log data and the difficulty in efficiently parsing it. This method, implemented on a general-purpose computing platform, uses log semantic vector clustering as a foundation, combined with template normalization and variable reverse extraction, to achieve high-quality structured modeling of log data.

[0008] The structured parsing and semantic template normalization method for unmanned vehicle operation logs of the present invention includes log normalization processing, semantic clustering based on a multilingual sentence vector model, log template normalization and variable extraction, template semantic and structural similarity analysis and merging, and reverse variable extraction and structured log reconstruction based on the merged template. The method comprises the following steps:

[0009] Step 1: Obtain the original unmanned vehicle operation log and preprocess it, including removing duplicate logs and structuring key fields. The processed log is then output in JSON format as input for subsequent processing.

[0010] Step 2: Extract the message field text from the structured log and convert it into a semantic vector. Use the multilingual sentence vector encoding model (paraphrase-multilingual-mpnet-base-v2) to represent each log entry as a 768-dimensional semantic vector. Then use incremental principal component analysis (PCA) to reduce the vector dimension to 50 dimensions.

[0011] Step 3: Use the Mini-Batch K-Means clustering algorithm to cluster the reduced vectors, dividing the logs into clusters based on semantic similarity. Each log is assigned a cluster ID (cluster_id), which is used as the grouping basis in the subsequent template generation process.

[0012] Step 4: Group the clustered logs according to their cluster numbers and normalize each group of logs. Use regular expressions to sequentially extract variable fields such as numbers, IP addresses, paths, identifiers, and time from the logs. Replace these fields with placeholders to generate a preliminary standardized template. De-duplicate the normalized templates, retaining only unique template styles within each group. Assign each template a template number. Record the mapping between the logs and the template numbers, extract the replaced fields, and output them as a variable list.

[0013] Step 5: Perform similarity merging on the initially generated templates. Use the multilingual sentence vector encoding model to represent the semantic vectors of the initially generated templates, and calculate the cosine similarity and edit distance between the templates. Multiple templates that meet the merging criteria, provided they meet the semantic similarity and edit distance thresholds, are grouped into the same template cluster, and the template closest to the center vector is selected as the representative template. A template number is assigned to the new template, and a mapping table is constructed from the original template number to the new template number.

[0014] Step 6: Based on the template number mapping results, rematch the original log and reversely extract variables. Use the merged representative template to generate a regular expression, reapply it to the original log content, extract the corresponding variable values, and record them in variable order. Finally, a structured log record containing fields such as log time, thread name, log level, class name, method name, template number, template content, and variable list is constructed and output in CSV format to provide input for subsequent analysis and anomaly detection modules.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. This invention utilizes a "clustering first, normalization second" processing sequence, avoiding the potential imbalance in template granularity caused by direct normalization in traditional methods. Experimental verification shows that compared to the 65% template extraction accuracy of the Drain algorithm, this invention achieves an accuracy rate exceeding 85%, effectively solving the template recognition challenge in complex, nested logs for autonomous vehicles.

[0017] 2. By introducing a dual constraint mechanism of semantic similarity and edit distance, we achieve precise merging of similar templates, reducing template redundancy by over 30% while maintaining over 95% semantic accuracy. The processing time for a single log entry is controlled within 0.1ms, meeting real-time processing requirements.

[0018] 3. Combining template normalization processing with reverse variable extraction technology, the variable extraction completeness rate reaches over 95%, a 15% improvement over traditional regular expression methods, providing a reliable data foundation for subsequent log analysis and anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The overall architecture diagram of the structured parsing and semantic template normalization method for unmanned vehicle operation logs

[0020] Figure 2 Flowchart of semantic feature extraction and dimensionality reduction based on multilingual sentence vector encoding

[0021] Figure 3 Flowchart for semantic clustering and template extraction

[0022] Figure 4 Flowchart of template merging mechanism based on semantic similarity DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings.

[0024] The present invention relates to a structured analysis method for unmanned vehicle operation logs based on semantic clustering and template normalization. The architecture process is as follows: Figure 1 As shown, it includes the following steps:

[0025] Step 1: After obtaining the original unmanned vehicle operation log, perform preprocessing operations on it, including the following three steps:

[0026] 1) Deduplication: The system uses a hash set construction mechanism to perform a full scan and duplicate detection of log content. For each log record, if the original string does not appear before, it will be retained and included in subsequent processing; if duplicates with identical content are detected, they will be automatically removed.

[0027] 2) Field Parsing: Based on predefined regular expression rules, the system parses the original log text after deduplication, extracts key field information, and completes preliminary semantic mapping. Log field extraction follows a unified field specification, covering six types of information: timestamp, thread name, log level, class name, method name, and log body content. The field definitions are shown in Table A.

[0028] Table A Log Field Extraction and Definition Specifications

[0029]

[0030] 3) Structured Output: The extracted fields are organized into a standardized structured data format. The system stores the field contents of each log record in JSON format, completing the initial conversion of logs from text to structured data.

[0031] Step 2: Extract the message field from each structured log and use the multilingual sentence vector encoding model to perform semantic encoding on it, converting it into a semantic vector with a dimension of 768. Let the i-th log be message i , then its corresponding embedding vector is expressed as:

[0032] v i =f(message i ), where f:S→R 768

[0033] Among them, s represents the string space, f is the sentence vector encoding function, V i ∈R 768 represents the 768-dimensional semantic vector corresponding to the i-th log. The above embedding process can capture the semantic features of the log content, making log records with similar semantics have a smaller Euclidean distance in the vector space.

[0034] Then, the incremental principal component analysis method is used to reduce the dimensionality of the above 768-dimensional semantic vector, with a target dimension of 50. First, all semantic vectors are unified into float32 type to reduce memory usage, and then the covariance matrix C of the semantic vector is fitted in batches. The calculation formula is:

[0035]

[0036] Where μ is the sample mean vector.

[0037] Then, the covariance matrix C is decomposed into eigenvalues, and the first 50 principal component directions are extracted to form the dimension reduction transformation matrix W. The following linear transformation is performed on each original vector:

[0038] v' i =w T (v i -μ), v' i ∈R 50

[0039] Finally, we get a set of reduced dimensionality vectors The compression method used in this implementation can retain about 99.79% of the cumulative information. The choice of dimension 50 is based on the trade-off between information retention and computational efficiency: when the dimension is less than 40, the information loss exceeds 1%, and when the dimension is greater than 60, the computational overhead increases significantly while the information gain is limited. In actual deployment, it can effectively balance the semantic expression ability and computing resource overhead. The final generated will be used as log clustering and template generation in subsequent steps. The specific process of the above semantic encoding and dimensionality reduction processing is as follows: Figure 2 shown.

[0040] Step 3: This step is based on the small batch K-means clustering algorithm to reduce the dimension of the semantic vector set obtained in step 2. Clustering is performed to divide log samples into several semantic template candidate groups based on semantic feature similarity. The specific implementation process is as follows:

[0041] 1) Set the number of clusters K and the number of samples per batch B, and initialize each K log semantic center vector C = {c1, c2, ..., c k},c k ∈R 50 Based on the analysis of the characteristics of the unmanned vehicle log data, this embodiment sets K = 100 and B = 1024. The K value is determined by the elbow rule to ensure a balance between clustering effect and computational efficiency; the B value is set according to the system memory limit and processing efficiency requirements.

[0042] 2) For each semantic vector v i '∈R 50 , calculate the Euclidean distance between it and each semantic center vector. The calculation formula is as follows:

[0043]

[0044] 3) After each batch is processed, the semantic center vector of the corresponding cluster is adjusted according to the mean of the currently assigned logs;

[0045] 4) The clustering process will stop when any of the following conditions is met: the change in all cluster centers is less than the set threshold ε, or the number of iterations has reached the maximum number of steps T max ;

[0046] 5) Finally, each log is assigned a cluster number (i.e., cluster_id), which is used as the grouping basis in the log template extraction stage, thereby achieving preliminary classification of templates at the semantic level.

[0047] like Figure 3 As shown in the figure, the clustering module performs small-batch K-means clustering based on the semantic vector after dimensionality reduction, outputs the template cluster number of each log, and passes the clustering result to the template extraction module to implement semantic-driven template classification processing.

[0048] Step 4: Based on the cluster ID (cluster_id) obtained in step 3, all log records are grouped according to the semantic cluster to which they belong. For each log set in the cluster, a set of predefined regular expression templates are applied to its message field in turn to extract variable fields in the log, including but not limited to: vehicle number, IP address, timestamp, path information, resource identifier and other dynamic elements. Each type of variable field is uniformly replaced with a semantically identifying placeholder, such as<Vehicle_id> 、 <ip> 、 <timestamp>Etc., and then generate a standardized initial template with a unified structure.

[0049] Then, the standardized log text after placeholder replacement in each cluster is deduplicated, and the template with unique structure is retained to obtain the preliminary template set T = {T1, T2, ..., T m }, T i ∈S', where S' represents the template set space formed by field extraction and placeholder replacement. Each template in the template set is assigned a unique template number (template_id) and a log template dictionary table is constructed. For each original log record log_id, the following two mapping relationships are established:

[0050] 1) log_id→template_id: records the standardized template number to which the log log_id belongs;

[0051] 2) log_id → variable_list: records the variable field contents replaced from the log log_id, and forms a variable list in the order of their appearance in the log.

[0052] Finally, two structured result files are output:

[0053] 1) Structured.csv: Contains the fields log_id, template_id, template_text (template text), variable_list (variable list), and origin_log (original log content);

[0054] 2) Template.csv: Contains the fields template_id and the corresponding template_text, forming the template dictionary table.

[0055] The following is a specific example to illustrate the processing process of this step:

[0056] The message field of the log: vehicle MLCB1D00039 start to check command execute;

[0057] Standardized template after regular expression processing: vehicle<Vehicle_id> start to checkcommand ex ecute; Extracted variable list: [Vehicle_id:MLCB1D00039].

[0058] Step 5: After completing the initial template extraction, in order to reduce the semantic redundancy and structural duplication in the template set, a dual constraint mechanism of semantic similarity and structural edit distance is used to perform fusion aggregation on all extracted templates. First, a multilingual sentence vector model is used to embed each template text T i Perform semantic encoding and generate the corresponding high-dimensional vector representation: v' i ∈R d , the vector dimension d is 768. Then, for each pair of templates (T i , T j ), first calculate the cosine similarity (CosineSimilarity):

[0059]

[0060] Then calculate the structural edit distance (Levenshtein distance):

[0061]

[0062] When a pair of templates simultaneously meet:

[0063] Sim cos ≧θ s =0.988, Dist edit ≦θ e =0.15

[0064] It is considered that they are highly similar at both the semantic and structural levels and can be classified into the same template cluster (template_cluster). The above threshold is determined by experimental tuning of 1000 unmanned vehicle log samples: θ s =0.988 can avoid excessive merging while ensuring the semantic consistency of the template; θ e =0.15 is based on the average length of unmanned vehicle logs and the statistical distribution characteristics of variables.

[0065] For each formed template cluster G, the system calculates the vector average of all its member templates as the cluster center. The calculation formula is as follows:

[0066]

[0067] And select the template closest to the cluster center as the representative template T of the cluster * , and its selection criteria are:

[0068]

[0069] The final selected representative template is considered the merged template and is uniformly assigned a new template number, new_template_id. The system also creates a mapping table from the original template number, template_id, to the new number, new_template_id, for unified replacement and management of template numbers in subsequent log records. All selected representative templates constitute a new template set, which the system enters into the file, Template_after_merge.csv. The field structure in this file is consistent with the fields in the Template.csv file generated in step 4, including:

[0070] 1) template_id: the template ID after the merge;

[0071] 2) template_text: corresponds to the template text.

[0072] like Figure 4 As shown in the figure, the template merging optimization mechanism realizes the intelligent aggregation of templates and the automatic selection of representative templates through the dual constraints of semantic similarity and edit distance.

[0073] Step 6: After the template number mapping relationship is established, the system uniformly updates and reconstructs the original structured log data. The processing flow is as follows:

[0074] 1) Based on the mapping table generated in step 5, the system replaces the template_id field in each log record with the corresponding merged ID new_template_id. For log records whose template ID and text remain unchanged during the merge process, the system directly reuses the previous variable extraction results to avoid repeated calculations. For log records whose templates have changed, the system re-executes the variable extraction operation to ensure that they are consistent with the new template structure;

[0075] 2) For each new_template_id corresponding to the representative template T * , the system automatically generates the corresponding regular expression R * , change the variable placeholder in the template (such as<Vehicle_id> 、 <ip>) is replaced with a regular expression to match the field value of the corresponding type. Apply the regular expression R to the log text that needs to be re-extracted * Perform matching operations and extract the variable list variable_list;

[0076] 3) All processed log records are structured and output to the file structured_after_merger.csv. The fields in this file follow the standard structured.csv definition from step 4, including log number, template number, template text, variable list, and original log content. However, unlike step 4, the template number and template text used in structured_after_merger.csv are the template number (i.e., new_template_id) and template text after the merge.< / ip> < / timestamp> < / ip>

Claims

1. A method for structured parsing and semantic template normalization of unmanned vehicle operation logs, characterized by: The steps include: Step 1: Obtain the original unmanned vehicle operation log and preprocess it, including removing duplicate logs and structuring key fields. The processed log is then output in JSON format as input for subsequent processing. Step 2: Extract the message field text from the structured log and convert it into a semantic vector. Use the multilingual sentence vector encoding model (paraphrase-multilingual-mpnet-base-v2) to represent each log entry as a 768-dimensional semantic vector. Then use incremental principal component analysis (PCA) to reduce the vector dimension to 50 dimensions. Step 3: Use the Mini-Batch K-Means clustering algorithm to cluster the reduced vectors, dividing the logs into clusters based on semantic similarity. Each log is assigned a cluster ID (cluster_id), which is used as the grouping basis in the subsequent template generation process. Step 4: Group the clustered logs according to their cluster numbers and normalize each group of logs. Use regular expressions to sequentially extract variable fields such as numbers, IP addresses, paths, identifiers, and time from the logs. Replace these fields with placeholders to generate a preliminary standardized template. De-duplicate the normalized templates, retaining only unique template styles within each group. Assign each template a template number. Record the mapping between the logs and the template numbers, extract the replaced fields, and output them as a variable list. Step 5: Perform similarity merging processing on the initially generated templates. The preliminarily generated templates are represented as semantic vectors using a multilingual sentence vector encoding model. The cosine similarity and edit distance between the templates are calculated. Multiple templates that meet the merging criteria, provided that the semantic similarity and edit distance thresholds are met, are grouped into the same template cluster. The template closest to the center vector is selected as the representative template. The new template is assigned a template number, and a mapping table is constructed from the original template number to the new template number. Step 6: Based on the template number mapping results, rematch the original log and reversely extract variables. Use the merged representative template to generate a regular expression, reapply it to the original log content, extract the corresponding variable values, and record them in variable order. Finally, a structured log record containing fields such as log time, thread name, log level, class name, method name, template number, template content, and variable list is constructed and output in CSV format to provide input for subsequent analysis and anomaly detection modules.

2. According to claim 1, extracting message field text from structured logs and converting it into a semantic vector. It is characterized in that: It includes a semantic feature extraction module based on multilingual sentence vector encoding: the semantic feature extraction module uses the multilingual sentence vector encoding model to perform semantic vectorization processing on the log message field; The encoding model receives the message field text in the structured log as input, and each log text is processed by the encoding model to output a 768-dimensional semantic vector representation; To reduce computational complexity and storage requirements, the incremental principal component analysis method is used to reduce the dimensionality of the 768-dimensional semantic vector, compressing the vector dimension to 50 dimensions while retaining the main semantic feature information. The 50-dimensional semantic vector after dimensionality reduction retains the main features of the original semantic information and improves the efficiency of subsequent clustering calculations.

3. According to claim 1, the mini-batch K-means clustering algorithm is used to perform clustering calculations on the vector set after dimensionality reduction, and the logs are divided into several clusters according to semantic similarity. It is characterized in that: A small-batch K-means clustering algorithm is used to automatically group semantically similar logs. The algorithm performs clustering calculations based on the 50-dimensional semantic vector after dimensionality reduction, and automatically divides logs into several semantically similar clusters based on semantic similarity. By processing in small batches, the system can efficiently process large amounts of log data without manually pre-setting template rules, and can adaptively discover patterns in logs. Each log is automatically assigned a cluster number, which is used as the grouping basis for subsequent template generation and normalization processing.

4. The method of performing similarity merging processing on the initially generated templates according to claim 1, characterized in that: The template merging process includes: using a multilingual sentence vector encoding model to generate a semantic vector for each initial template, and calculating the cosine similarity between the template vectors; The Levenshtein edit distance algorithm is used to calculate the degree of difference between template strings; When the cosine similarity is greater than 0.988 and the edit distance is less than 15% of the template length, the template is marked as mergeable; The average value of all vectors in each template cluster is calculated as the cluster center, and the template closest to the cluster center is selected as the representative template.

Images