ASC log analysis system and method based on multimode CAN bus protocol

Abstract

The invention discloses an ASC log analysis system and method based on a multimode CAN bus protocol, and the system comprises an obtaining module, an analysis module and a storage module, the obtaining module is used for obtaining an ASC log file, the analysis module is used for analyzing the format of the ASC log file, verifying the ASC log file, carrying out the standardized field conversion of the ASC log file, generating an analysis result list, and storing the analysis result list in the storage module. And the storage module is used for configuring a data storage environment, generating a database, inserting the data of the analysis result list into the database in batches, performing transaction management and counting the data increment of the analysis result list. The method can be compatible with analysis of the ASC logs in multiple formats, the large file processing efficiency is greatly improved, and the query time for positioning a specific CAN ID or a time period is shortened.

Description

Technical Field

[0001] This invention relates to the field of automotive electronic control technology, specifically to an ASC log parsing system and method based on the multi-mode CAN bus protocol. Background Technology

[0002] CAN bus log analysis is a crucial step in the development and fault diagnosis of automotive electronic control systems. Current technologies face the following bottlenecks: Insufficient format compatibility: The ASC log formats generated by different CAN analysis devices vary, including standard format, extended frame format, compact format and other variants, which traditional parsers cannot fully support; Large file processing efficiency is low: CAN log files generated during vehicle road tests are usually hundreds of MB or even GB in size. The traditional full memory loading method leads to slow system response or even crashes. Poor data retrieval performance: The linear scanning query method takes too long to locate a specific CAN ID or signal within a large amount of data; Lack of cache management mechanism: When repeatedly querying the same signal data, there is a lack of an effective cache reuse mechanism, resulting in a waste of computing resources. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention provides an ASC log parsing system and method based on the multi-mode CAN bus protocol, which is compatible with the parsing of multiple ASC log formats, significantly improves the efficiency of large file processing, and reduces the query time for locating specific CAN IDs or time periods.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to a first aspect of this application, an ASC log parsing system based on a multi-mode CAN bus protocol is provided, comprising: The file acquisition module is used to acquire ASC log files; The parsing module includes a regular expression list creation submodule, a validation criterion creation submodule, and a standardization submodule. The regular expression list creation submodule is used to parse the format of the ASC log file, the validation criterion creation submodule is used to validate the ASC log file, and the standardization submodule is used to perform standardization field transformation on the ASC log file and generate a parsing result list. The storage module includes a database submodule and a batch insertion and transaction management submodule. The database submodule is used to configure the data storage environment and generate the database. The batch insertion and transaction management submodule is used to batch insert the data of the parsed result list into the database and perform transaction management.

[0006] In some embodiments of this application, based on the aforementioned scheme, the regular expression list building submodule matches the format of the ASC log file using a regular expression list. The regular expression list includes regular expressions that are mapped one-to-one with the format of the ASC log file. When performing the matching, each valid line of the ASC log file is processed, and the regular expressions are matched in a loop according to their order. The order is based on the frequency of the format in the actual scenario. After a successful match, the core fields are extracted according to the matched format, and the matching and verification of the subsequent formats are terminated.

[0007] In some embodiments of this application, based on the foregoing scheme, the verification criteria establishment submodule verifies the ASCII log file by establishing verification criteria, which include existence verification criteria, permission verification criteria, encoding verification criteria, and large file adaptation criteria; and / or, The standardization submodule includes: A filtering unit is used to filter invalid lines, including blank lines, comment lines, and lines with malformed formatting. The correction unit is used to remove leading and trailing whitespace, replace full-width spaces, unify capitalization, and unify separators.

[0008] In some embodiments of this application, based on the foregoing scheme, the database submodule includes: The database connection configuration unit is used to support multi-threaded shared connections, enable pre-write logging mode, support read and write concurrency, reduce synchronization level, set memory mapping, and set caching. The table structure creation unit is used to create a raw data table and a statistical data table. The raw data table is used to store the complete data of each parsed item in the parsing result list; the statistical data table is used to store the aggregated statistical information of each CAN_ID. The index optimization sub-unit is used to create composite indexes, which filter data based on specific channels and directions for high-frequency query scenarios.

[0009] In some embodiments of this application, based on the foregoing scheme, the batch insertion and transaction management submodule includes: The batching logic unit is used to split the parsing result list, divide the parsing result list into several batches, and implement batch processing. The transaction execution unit is used to call placeholder functions, batch execution functions, transaction guarantee functions, and field mapping verification functions for each batch of data. The placeholder functions are used to replace dynamic values ​​in the SQL statement. The batch execution functions are used to insert data from the parsed result list into the database in batches. The transaction guarantee functions are used to put the batch execution functions into the transaction management scope. If the insertion is successful, the transaction is committed. If an exception occurs, the transaction is rolled back. The field mapping verification function is used to adjust the core fields in the parsed result list to be strictly consistent with the field order of the SQL insertion statement, and to convert them into a list of tuples through list comprehensions.

[0010] In some embodiments of this application, based on the foregoing scheme, a query module is further included, the query module comprising: The condition generation unit is used to generate cached items and unique cache keys for query conditions; The linked query unit is used to verify the validity of the cache, including key existence verification and expiration time verification. The key existence verification is used to check whether the cache key exists in the preset cache dictionary. If the key does not exist, it is determined as a cache miss. If the key exists, the interval between the current time and the creation time is calculated by the expiration time verification. If it is greater than the cache expiration time, the cache item is determined to be expired, the cache item is deleted and determined as a cache miss. If it is less than or equal to the cache expiration time, the cache item is determined to be valid and the query result is obtained.

[0011] In some embodiments of this application, based on the foregoing scheme, when determining that the cached item is valid, obtaining the query result further includes: Update the access records, which include the number of accesses and the access time. If the list length exceeds 2, delete the oldest access record. Return the cached items for the query criteria; and / or, The process of determining that a cached item is expired, deleting the cached item, and determining a cache miss also includes: Query the database and filter query conditions by dynamically concatenating SQL statements; Obtain the field names of the filtered query conditions, and convert the list of tuples returned by the query into a dictionary list, ensuring that the format is consistent with the list of parsed results; If the query result of the query condition is not empty, write the query result of the query condition to the cache and trigger the cache eviction logic at the same time. Returns the query results based on the database query conditions.

[0012] According to a second aspect of this application, an ASC log parsing method based on a multi-mode CAN bus protocol is provided, including: Get the ASC log file; Parse and validate the format of the ASC log file, perform standardized field transformation on the ASC log file, and generate a list of parsing results; Configure the data storage environment and generate the database; The data from the parsed result list is inserted into the database in batches, and transaction management is performed. The data increment of the parsed result list is also counted. Generate cached items and unique cache keys for the query conditions, and execute the query task.

[0013] According to a third aspect of this application, a computer-readable storage medium is provided that stores a computer program thereon, the computer program including executable instructions that, when executed by a processor, implement the method described above.

[0014] According to a fourth aspect of this application, an electronic device is provided, comprising: One or more processors; A memory for storing executable instructions of the processor, which, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0015] The beneficial effects of this application are as follows: (1) The ASC log parsing system and method based on the multi-mode CAN bus protocol provided in this application clearly defines the core objectives of the system as multi-format ASC parsing, efficient storage and fast query. Based on the principle of single responsibility, it is divided into three core modules (parsing module, storage module and query module), and combined with the file acquisition module, it forms a multi-module architecture to ensure low coupling and high cohesion between modules.

[0016] (2) The ASC log parsing system and method based on the multi-mode CAN bus protocol provided in this application are developed in a global configuration center, establish a regular expression list to centrally manage regular expressions, establish database parameters such as connection path, batch insertion threshold, optimization parameters, and establish caching strategies (such as capacity, expiration time, and eviction weight to avoid hard coding), and provide unified parameter support for subsequent module development.

[0017] (3) The ASC log parsing system and method based on the multi-mode CAN bus protocol provided in this application are developed for multi-format parsing. First, the line preprocessing logic is implemented, then the regular expression is pre-compiled to improve the matching efficiency. After matching 7 ASC formats according to priority, the data is standardized and finally the file batch parsing method is encapsulated.

[0018] (4) The ASC log parsing system and method based on the multi-mode CAN bus protocol provided in this application are used to develop the database storage module: first, the database is initialized, such as creating the original data table, statistical data table, and composite index, configuring performance optimization parameters, and then the three core functions of batch insertion, incremental statistics, and conditional query are implemented to ensure efficient and consistent data storage.

[0019] (5) The ASC log parsing system and method based on the multi-mode CAN bus protocol provided in this application uses multi-level cache development as the core of query performance optimization. First, the cache storage structure is designed to realize cache key generation, priority score calculation and intelligent eviction logic. Then, the cache add, delete, modify and query interface is encapsulated to reduce the pressure of direct database access.

[0020] (6) The ASC log parsing system and method based on the multi-mode CAN bus protocol provided in this application connects the entire process. After initializing the three core modules, the process is executed according to the following steps: parsing ASC files, storing them in the database, cache query linkage, and resource release execution. Independent test code for each module is written to verify the correctness of the module functions, and finally a complete and usable system is formed.

[0021] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and are intended to explain the invention, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the ASC log parsing system based on the multi-mode CAN bus protocol of the present invention; Figure 2 This is a schematic diagram of the parsing module of the present invention; Figure 3 This is a schematic diagram of the storage module of the present invention; Figure 4 This is a schematic diagram of the ASC log parsing method based on the multi-mode CAN bus protocol of the present invention; Figure 5 This is a schematic diagram of an electronic device according to the present invention. Detailed Implementation

[0023] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0024] It should be understood that the terms "comprising" and other similar expressions in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, or apparatus that includes a series of steps or units and is not limited to the listed steps or units. Furthermore, "first" and "second" are used to distinguish different objects and are not intended to describe a specific order.

[0025] According to the first aspect of this application, Figures 1-3 As shown, this embodiment provides an ASC log parsing system based on the multi-mode CAN bus protocol, including: The file acquisition module is used to acquire ASC log files; The parsing module includes a regular expression list creation submodule, a validation criterion creation submodule, and a standardization submodule. The regular expression list creation submodule is used to parse the format of the ASC log file, the validation criterion creation submodule is used to validate the ASC log file, and the standardization submodule is used to perform standardization field transformation on the ASC log file and generate a parsing result list. The storage module includes a database submodule and a batch insertion and transaction management submodule. The database submodule is used to configure the data storage environment and generate the database. The batch insertion and transaction management submodule is used to batch insert the data of the parsed result list into the database and perform transaction management. The query module is used to generate cached items for query conditions and unique cache keys corresponding to the query conditions, and to execute query tasks.

[0026] Thus, the ASC log parsing system based on the multi-mode CAN bus protocol provided in this embodiment combines multi-format ASC parsing, efficient storage, and fast querying. Based on the principle of single responsibility, it is divided into three core modules (parsing module, storage module, and query module), which, together with the file acquisition module, ensure low coupling and high cohesion between modules.

[0027] The following section uses Python function code as an example to provide a detailed description of the ASC log parsing system based on the multimode CAN bus protocol provided in this embodiment.

[0028] Parameter linkage feature: Key parameters remain linked across multiple modules to ensure process consistency. For example, Config.BATCH_INSERT_SIZE (default 50,000 records) simultaneously affects: The storage module's _batch_insert_raw_data() function is used to split batches of inserted data. The save_data() call logic in the entry program (main.py) ensures consistency in batch processing. In some embodiments of this example, the ASCII log file is a CAN bus communication record file in a specific format. It is a vehicle CAN network data log stored in ASCII text form and is an important carrier for vehicle electronic system diagnosis and data analysis.

[0029] Specifically, the file acquisition module can read ASC log files generated by different CAN analysis devices. Therefore, the acquired ASC log files are ASC log files under the multi-mode CAN bus protocol.

[0030] In some implementations of this embodiment, a regular expression list creation submodule is used to create a regular expression list and match the format of the ASCII log file using the regular expression list. The format includes standard frames, extended frames, compact frames, error frames, remote frames, square bracket frames, etc.

[0031] In some specific implementations, the regular expression list predefines multiple formats, and the specific matching relationships are as follows: A standard frame (such as "1.2345 123 Tx d 8 00 01") matches the first regular expression; Extended frames (such as "1.2345 123 Tx x 8 00 01") match the 7th regular expression; Other formats (compact frames, error frames, remote frames, bracket frames, extended identifier frames, etc.) are matched according to the order of the regular expression list.

[0032] In some implementations of this embodiment, parsing efficiency and accuracy are improved through regular expression pre-compilation and priority matching mechanisms.

[0033] Regular expression pre-compilation optimization: During initialization, the seven formats of regular expressions are compiled only once, avoiding repeated compilation during parsing of each line. Tests have shown that this can improve parsing efficiency by more than 30%.

[0034] Matching execution logic: When processing a single valid line, it performs a cyclical match in the following order (standard frame, extended frame, compact frame, error frame, remote frame, bracket frame, extended identifier frame). This order is based on the frequency of format occurrence in the actual scenario to improve the match hit rate. After a successful match, the core fields are extracted and the result is returned immediately, terminating the matching and validation of subsequent formats (for example, after the first regular expression in the standard frame is matched, the subsequent 6 regular expressions are no longer validated).

[0035] Add format identifier: Add a format type identifier to the parsed result based on the regular expression index that is successfully matched by idx. For example, idx=0 corresponds to "standard frame" and idx=3 corresponds to "error frame format". This identifier is stored in the "format_type" format type field of the returned dictionary (this can be implemented in the code through extensions to facilitate subsequent data classification and statistics).

[0036] In some implementations of this embodiment, fields are precisely extracted based on regular expression lists, and the consistency and validity of the output data are ensured through standardized processing and validation criteria.

[0037] Core field extraction: Based on regular expressions of different formats, the index is grouped to accurately extract core fields such as timestamp, channel number, CAN_ID, direction, and data. The extraction logic for key formats is as follows: Standard frame (idx=0): extract timestamp using match.group(1), channel number using match.group(2), CAN_ID using match.group(3), direction using match.group(4), and data using match.group(6); Compact frame (idx=2): The data field is a continuous string without spaces (such as "00010203"). It needs to be split into 2 characters and spaces added by using .join([raw_data[i:i+2]foriinrange(0,len(raw_data),2)]) to uniformly convert it into the format "00010203". Square bracket frame (idx=5): Data fields are enclosed in square brackets (e.g., "[000102]"). The content inside the brackets needs to be extracted directly and spaces removed to get "000102".

[0038] Perform standardization processing on the fields of the ASC log file, and simultaneously validate the validity of the ASC log file: Timestamp: Converted to floating-point type using float(match.group(1)), retaining 6 decimal places (e.g., "1.2345"-1.234500), checking for non-negativity, and marking negative numbers as parsing failure; Channel number: Convert to an integer using int(match.group(2)), the range is 1-255, and if it exceeds the range, it is marked as parsing failure; CAN_ID: Convert CAN_ID to lowercase string and check if CAN_ID is a hexadecimal character. If it contains non-hexadecimal characters, mark it as a failure. Direction: Use match.group(4).upper() to convert to uppercase (e.g., "tx" to "TX", "rx" to "RX"). Error frames and remote frames have no direction field, so set them to an empty string.

[0039] Exception handling mechanism: When parsing a single piece of data fails (no matching format or field validation fails), the exception is caught by try-except and recorded in the log as "Parsing of line {line_num} failed: {exception reason} - original line: {line}", for example, "Parsing of line 100 failed: CAN_ID contains non-hexadecimal characters - original line: 1.2345123Txd80g01". Failed data is not included in the output and does not affect the overall parsing process.

[0040] In some implementations of this embodiment, a verification criteria establishment submodule is used to establish verification criteria, which include existence verification criteria, permission verification criteria, encoding verification criteria, and ultra-large file adaptation criteria.

[0041] In some specific implementations, the verification criteria establishment submodule verifies the validity of the ASC log file upon startup, specifically including: Existence checks are performed using existence verification criteria to verify the existence of the ASC log file. Permissions are verified using permission validation criteria to confirm whether the user has read permissions for the ASC log file. Encoding verification is performed using encoding verification criteria to force UTF-8 encoding for reading, thus avoiding garbled Chinese comments. If an encoding error is found, a decoding error is thrown and the process terminates.

[0042] Large files are adapted using a large file adaptation principle. For ASCII files larger than 1GB, a line-by-line streaming logic is triggered, loading only the current line into memory to avoid memory overflow issues caused by loading the entire file at once.

[0043] In some embodiments of this example, the standardization submodule includes a filtering unit and a correction unit, wherein the filtering unit is used to filter invalid rows and the correction unit is used to clean up fields.

[0044] In some embodiments of this example, the filtering unit is used to filter blank lines, comment lines, and lines with incorrect formatting.

[0045] In some specific implementations, the logic of the preprocess_line and parse_file functions in parser.py is used to filter invalid data and record traceability information by processing line by line.

[0046] Reading method: Call the parse_file() function of parser.py, adopt the "read line by line and process in real time" mode, and obtain the line number by for line_num, line in enumerate(f, 1) to provide a basis for subsequent exception tracing.

[0047] In some specific implementations, the filtering rules are executed as follows: filtering is performed sequentially in the order of "blank lines, comment lines, and lines with malformed formatting," with the specific logic as follows: Empty lines are marked as invalid lines by line.strip(); comment lines are identified by line.startswith(Config.INVALID_LINE_PREFIX) (i.e. " / / "); for inline comments (such as "1.2345 123 Tx / / test data"), the valid content before " / / " is extracted by line.split(" / / ")[0].strip(); for malformed lines that are missing key fields such as timestamps and channel numbers, they are temporarily stored and the parsing failure of line {line_num} is printed in the log; for malformed lines {line} that are missing key fields, the overall loop is not interrupted.

[0048] Statistical output: After preprocessing, the number of valid lines is counted based on the length of the parsing result list returned by parse_file(). Combined with the console log output, "Total number of parsed lines: xxx, number of successful parsings: xxx, number of failed parsings: xxx", where the number of failures includes the number of invalid lines filtered and the number of lines that failed to be parsed subsequently.

[0049] In some embodiments of this example, the correction unit is used to remove leading and trailing whitespace, replace full-width spaces, unify capitalization, and unify separators.

[0050] In one specific implementation, the following steps are performed: Leading and trailing whitespace are removed: whitespace characters such as spaces and newlines at the beginning and end of lines are removed using `line.strip()`; Full-width spaces are replaced: Chinese full-width spaces are converted to English half-width spaces using `line.replace`; Case is standardized: all letters are converted to lowercase using `line.lower()`, laying the groundwork for a unified lowercase format for CAN_ID; and delimiters are standardized using `re.sub(r"\s+", " ", line)`: irregular delimiters such as tabs and consecutive spaces are uniformly compressed into a single English space, ensuring consistent delimiters between fields and avoiding matching failures due to abnormal delimiters during regular expression matching.

[0051] In some embodiments of this example, the database submodule includes: The database connection configuration unit is used to support multi-threaded shared connections, enable pre-write logging mode, support read / write concurrency, reduce synchronization level, set memory mapping, and set caching.

[0052] The table structure creation unit is used to create a raw data table and a statistical data table. The raw data table (raw_can_data) stores the complete data of each parsed result in the parsed list, including the fields id (auto-incrementing primary key), timestamp, channel, can_id (CAN_ID), direction, data, and create_time. The statistical data table (can_statistics) stores the aggregated statistical information of each CAN_ID, including the fields can_id (primary key), total_count (total occurrences), last_data (recent data), first_timestamp (first occurrence timestamp), and last_timestamp (recent occurrence timestamp).

[0053] In some specific implementations, the database connection configuration unit establishes a multi-threaded shared connection, corresponding to the CANDataStorage._init_database logic, to complete database connection, table structure creation, and performance optimization configuration, providing a basic environment for data storage.

[0054] Database connection configuration: Connections are established via `sqlite3.connect(self.db_path, check_same_thread=False)`, where `check_same_thread=False` supports multi-threaded connection sharing, avoiding thread safety issues. Simultaneously, optimization parameters in `Config.DB_PRAGMAS` are executed: `journal_mode=WAL`: Enables pre-write logging mode, supporting concurrent read and write operations (default is only serial); `synchronous=NORMAL`: Reduces the synchronization level, decreasing disk I / O blocking and improving write speed; `mmap_size=268435456`: Sets 256MB of memory mapping, reducing disk reads during large file parsing; `cache_size=-10000`: Sets 10MB of cache (negative numbers represent KB), improving query efficiency.

[0055] In some specific implementations, the table structure creation unit also includes an index optimization subunit: creating a composite index, filtering by channel and direction to improve query speed for high-frequency query scenarios, and creating the index only once (skipping if it already exists).

[0056] In some implementations of this embodiment, the batch insertion and transaction management submodule improves write efficiency and ensures data consistency, including: The batching logic unit is used to split the parsing result list, divide the parsing result list into several batches and implement batch processing, so as to avoid memory overflow or IO blocking caused by excessive data insertion at one time.

[0057] The transaction execution unit is used to call placeholder functions, batch execution functions, transaction guarantee functions, and field mapping verification functions for each batch of data. The core logic is as follows: Placeholder functions: Replace dynamic values ​​in SQL statements with placeholders "?" to avoid SQL injection risks; Batch execution function: By performing inserts in batches, efficiency is improved by more than 50 times compared to single-row inserts; Transaction-guaranteed functions: Batch execution functions are placed within a "transaction management scope". If the batch insertion is successful, the transaction is committed. If an exception occurs, the transaction is rolled back, ensuring the atomicity of each batch of data (either all successes or all failures).

[0058] Field mapping validation function: The "timestamp", "channel", "can_id", "direction", and "data" keys in the parsed result list must strictly match the field order in the SQL insert statement. The function converts the data into a list of tuples using the list comprehension function values ​​= [(item["timestamp"], item["channel"], ...) for item in data_list] to ensure that the field mapping is correct.

[0059] In some embodiments of this example, the storage module further includes an incremental statistics submodule, which is used to count the data increment of the parsed result list.

[0060] In some implementations of this embodiment, the incremental statistics submodule corresponds to the CANDataStorage._update_statistics logic, which incrementally updates statistical information based on batch-inserted data, avoiding full calculation and improving efficiency.

[0061] In some implementations of this embodiment, the incremental statistics submodule aggregates statistical information by using a statistical aggregation function: grouping data by CAN_ID using the dictionary can_stat, and traversing batch data.

[0062] Specifically, the first appearance of CAN_ID: initializes the statistics, count=1, last_data=current data, first_ts=current timestamp, last_ts=current timestamp; For existing CAN_IDs: Update statistics, count += 1, last_data = current data, last_ts = current timestamp, the first timestamp (first_ts) remains unchanged.

[0063] UPSERT update operation: Executes the INSERT ... ON CONFLICT ... DO UPDATE statement (SQLite 3.24.0 and above), achieving efficient operation of "updating if it exists, inserting if it doesn't exist": Insertion logic: When CAN_ID does not exist, insert a new record and write the can_id, total_count, last_data, first_timestamp, and last_timestamp fields; Update logic: When CAN_ID already exists, execute total_count = total_count + excluded.total_count (cumulative count), last_data = excluded.last_data (update the most recent data), and last_timestamp = excluded.last_timestamp (update the most recent timestamp). The timestamp is not updated for the first time.

[0064] Results feedback: After each batch of statistical updates is completed, the log will print "Statistical update successful: Statistical information of xxx CAN_IDs updated". If an exception occurs, the transaction will be rolled back and "Statistical update failed: Exception information: {Exception details}" will be recorded.

[0065] In some implementations of this embodiment, the query module improves query response speed and ensures data accuracy through a linkage mechanism of "caching first, database as a backup". This is mainly achieved through the collaborative implementation of the query interfaces of cache.py and storage.py.

[0066] The corresponding CANDataCache._generate_key logic ensures the accuracy and consistency of cache queries through conditional standardization and unique cache key generation.

[0067] In some embodiments of this example, the query module includes: The condition generation unit is used to generate cached items and unique cache keys for query conditions. Specifically: Conditional standardization: When the encapsulated `query_can_data(channel, can_id)` function is called, standardization processing consistent with the parsing phase is automatically performed. Channel number: If the input is a string, it will attempt to convert it to an integer. If the conversion fails, it will return an empty list and prompt "Channel number must be an integer"; CAN_ID: Converted to lowercase using can_id.lower() to ensure consistency with the stored CAN_ID format; Timestamp range: If passed in, it will be converted to floating point. The start time will be checked to be less than or equal to the end time. Otherwise, the message "Invalid timestamp range" will be displayed.

[0068] Cache key uniqueness guarantee: A 32-bit MD5 hash cache key is generated using the _generate_key() function to ensure that different parameter combinations correspond to unique keys. Parameter sorting: Sort the standardized query parameters in ascending order by parameter name (sorted(kwargs.items(),key=lambda x: x[0])) to ensure that “channel=1&can_id=123” and “can_id=123&channel=1” generate the same key; Hash conversion: The sorted parameter string (format "key1=value1:key2=value2") is converted into a hash value using hashlib.md5(key_string.encode("utf-8")).hexdigest() to avoid storage efficiency issues caused by excessively long keys.

[0069] Cache item structure definition: Cache values ​​are of dictionary type, containing data and metadata.

[0070] The linked query unit is used to verify the validity of the cache, including key existence verification and expiration time verification. The key existence verification is used to check whether the cache key exists in the preset cache dictionary. If the key does not exist, it is determined as a cache miss. If the key exists, the interval between the current time and the creation time is calculated by the expiration time verification. If it is greater than the cache expiration time, the cache item is determined to be expired, the cache item is deleted and determined as a cache miss. If it is less than or equal to the cache expiration time, the cache item is determined to be valid and the query result is obtained.

[0071] In some specific implementations, the corresponding cache.get and storage.query_data collaborative logic is used to realize the complete process of "cached check hit returns if no hit, then writes the database to the cache".

[0072] Cache validation process: When calling cache.get(**cache_params), the cache validity is validated according to the following steps: Key existence check: Check if the cache key exists in the self.cache dictionary; if not, it is considered a cache miss. Expiration time check: If the key exists, calculate the interval between the current time and the creation time (current_time - create_time). If it is greater than the cache expiration time defined in the configuration (Config.CACHE_EXPIRE_SECONDS, default 3600 seconds), it is determined to be expired, the cache item is deleted and it is determined to be a cache miss.

[0073] Cache hit handling: When the cache is valid, perform the following operations and then return the data: Update access count: cache_entry["access_count"] += 1; Record access time: cache_entry["access_history"].append(current_time), if the list length exceeds 2, delete the earliest access record (cache_entry["access_history"].pop(0)), implement the LRU2 strategy; Returned data: Directly returns cache_entry["data"], without needing to access the database.

[0074] Cache miss handling: When the cache does not exist or has expired, execute the "query the database, write to the cache, and return the data" process. Database query: Call storage.query_data(channel, can_id) to achieve condition filtering by dynamically concatenating SQL statements. For example, when querying by channel, concatenate query_sql += " AND channel = ?". Use placeholder parameters (params.append(channel)) to avoid SQL injection. Result formatting: Get the field names by using columns = [desc[0] for desc in cursor.description], and convert the list of tuples returned by the query into a list of dictionaries ([dict(zip(columns, row)) for row incursor.fetchall()]) to ensure that the format is consistent with the parsed result; Write to cache: If the query result is not empty, call cache.set(data=db_result, **cache_params) to write the result to the cache and trigger the cache eviction logic at the same time; Return data: Returns the database query results to the caller.

[0075] Cache eviction mechanism: When the cache capacity exceeds Config.CACHE_MAX_SIZE (default 10,000 entries), the _evict_cache() function is triggered. Score Calculation: The priority score of each cached item is calculated using the _calculate_score() function, and a three-factor weighted sum is applied. Recent access score: weights["recent_access"] / (current time - last access time + 1), the more recent the access, the higher the score; Access frequency score: weights["access_count"] * number of visits; the more frequent the visits, the higher the score. Data size score: weights["data_size"] / (data size + 1), the smaller the data, the higher the score.

[0076] Evicting Execution: Sort by score in ascending order, delete the cached item with the lowest score, until the cache capacity is less than the maximum capacity of the cache. Print the log "Evicting cached item: key=xxx, score=xxx.xxxx".

[0077] Corresponding to the storage.query_data result formatting logic, ensuring that the output result format is consistent and supports diverse needs.

[0078] Consistent format: The output dictionary list contains fixed fields such as "timestamp", "channel", "can_id", "direction", and "data", which are completely consistent with the output format of the parsing stage, facilitating subsequent data processing.

[0079] Pagination support: When the query results exceed 1000, the pagination logic is extended in query_data(), and pagination is implemented through LIMIT ? OFFSET ?. The returned results include information such as "data list, total number of records, current page number, and total number of pages", for example, "data: [xxx], total number of records: 1200, current page: 1, total number of pages: 2".

[0080] Export functionality: Offers diverse export options based on query results; requires additional dependency import to implement. CSV export: Call pandas.DataFrame(result).to_csv("query_result.csv", index=False, encoding="utf-8"); JSON export: Call json.dumps(result, ensure_ascii=False, indent=2) and write to a file.

[0081] The query module performs resource cleanup and log archiving after the system finishes running, ensuring system stability and facilitating problem tracing. This corresponds to the cleanup logic in main.py and the closing interfaces of each module.

[0082] In some implementations of this example, a database resource release module is also included: calling the `storage.close()` method, and executing `self.conn.close()` within `CANDataStorage.close()` to close the database connection. A `finally` block ensures that even if the program is abnormally terminated, the cursor and connection are closed, preventing connection leaks.

[0083] In some embodiments of this example, the database resource release module further includes: Cache resource release unit: Call the cache.clear() method to clear the self.cache dictionary to release memory, print the log "Cache cleared: xxx cache items deleted", and at the same time calculate and output the cache hit rate (number of hits / total number of queries).

[0084] File resource release unit: All file operations (ASC file reading, log writing) are automatically closed using the with statement, eliminating the need for manual handling of file handles and ensuring complete data writing and thorough resource release.

[0085] In some embodiments of this example, a log content integration module is also included. This module is used to archive key logs throughout the entire process. These key logs include the following core metrics: Basic information: System startup time, shutdown time, runtime; Parsing statistics: parsed file path, total number of lines parsed, number of successful parsings, number of failed parsings, and line numbers of failed parsings; Storage statistics: total number of records inserted into the database, details of batch insertions, and the number of updated CAN_IDs; Query statistics: total number of queries, cache hits, cache hit rate, and number of results for each query condition; Error information: Details of file reading errors, parsing errors, database errors, cache errors, etc. that occur throughout the process.

[0086] In some implementations of this embodiment, the full-process critical log archiving rules are as follows: Naming convention: Log files should be named in the format "date_time.log", for example, "20250601_143000.log"; Storage period: By default, log files from the most recent 30 days are retained. Expired logs are cleaned up via scheduled scripts or manually. Storage path: Log files are uniformly stored in the "logs" folder in the project root directory for easy centralized management.

[0087] Final report output: When the program ends, a full-process statistical report will be output to both the console and the log file, as shown in the example below: In one specific embodiment, the CAN bus multi-mode parsing system based on the multi-mode CAN bus protocol disclosed in this embodiment is used. The CAN bus multi-mode parsing system operation report shows: Startup time: 2025-06-01 14:00:00; End time: 2025-06-01 14:05:30; Runtime: 330 seconds. Parsing statistics: File path = test_can.asc, Total lines = 100000, Success = 98500, Failure = 1500. Storage statistics: Total number of inserted records = 98500, completed in 10 batches; Number of updated CAN_IDs = 200. Query statistics: Total number of queries = 10, Cache hit = 8, Hit rate = 80%. Anomaly information: No serious anomalies; 1500 parsing failures (see log details).

[0088] Thus, this application clearly defines the core objectives of the system as multi-format ASCII parsing, efficient storage, and fast querying. Based on the principle of single responsibility, it breaks down the system into three core modules (parsing module, storage module, and query module), which, together with the file acquisition module, form a multi-module architecture to ensure low coupling and high cohesion between modules.

[0089] This application develops a global configuration center, establishing a regular expression list to centrally manage regular expressions, and establishing database parameters (connection path, batch insertion threshold, optimization parameters) and caching strategies (capacity, expiration time, eviction weight) to avoid hard coding and provide unified parameter support for subsequent module development.

[0090] This application develops a multi-format parsing method. First, it implements preprocessing logic (filtering comments / blank lines), then precompiles regular expressions to improve matching efficiency, matches 7 ASCII formats according to priority, and then performs data standardization processing (unifying CAN_ID case, timestamp format, etc.). Finally, it encapsulates a batch file parsing method.

[0091] This application provides a database storage module development approach: first, initialize the database (create raw data tables, statistical data tables, and composite indexes, and configure performance optimization parameters), and then implement three core functions: batch insertion (transaction wrapping and executemany to reduce IO), incremental statistics (UPSERT to avoid full calculations), and conditional queries (dynamically concatenated SQL to prevent injection), ensuring efficient and consistent data storage.

[0092] This application develops a multi-level caching module: caching.py is developed as the core for query performance optimization. First, the cache storage structure (MD5 unique key and cache item metadata) is designed, and the logic for cache key generation (parameter sorting and hashing to ensure uniqueness), priority score calculation (three-factor weighting), and intelligent eviction (LRU2 and score sorting) is implemented. Then, the cache CRUD interface is encapsulated to reduce the pressure of direct database access.

[0093] This application connects the entire process, initializes the three core modules, and then executes the process according to parsing ASCII files, storing them in the database, cache query linkage, and resource release. Independent test code is written for each module (such as the parser's seven format tests and cache eviction / expiration tests) to verify the correctness of module functions, and finally form a complete and usable system.

[0094] According to the second aspect of this application, such as Figure 4 As shown, the ASCII log parsing method based on the multi-mode CAN bus protocol provided in this embodiment includes: Step S1: Obtain the ASCII log file; Step S2: Parse and verify the format of the ASC log file, and perform standardized field transformation on the ASC log file to generate a list of parsing results; Step S3: Configure the data storage environment and generate the database; Step S4: Batch insert the data from the parsed result list into the database and perform transaction management, and count the data increment of the parsed result list; Step S5: Generate cache items for the query conditions and unique cache keys corresponding to the query conditions, and execute the query task.

[0095] Specifically, this embodiment corresponds one-to-one with the above system embodiment. The functions of each step have been described in detail in the corresponding method embodiment, so they will not be repeated here.

[0096] According to a third aspect of this application, this embodiment provides a computer-readable storage medium having a computer program stored thereon, the computer program including executable instructions that, when executed by a processor, implement the method described above.

[0097] The present invention can implement all or part of the processes in the above methods, or it can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0098] According to the fourth aspect of this application, such as Figure 5 As shown, an electronic device is provided, comprising: One or more processors; Memory is used to store executable instructions for the processor, which, when executed by one or more processors, cause one or more processors to implement the methods described above.

[0099] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and a bus connecting different system components (including memory and processor).

[0100] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of a computer system, connecting all parts of the computer system through various interfaces and lines.

[0101] Memory can be used to store computer programs and / or modules. The processor implements various functions of the computer system by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system and at least one application program required for a function (e.g., sound playback, image playback, etc.); the data storage area can store data created based on the use of the mobile phone (e.g., audio data, video data, etc.). Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SmartMedia Cards (SMC), Secure Digital (SD) cards, Flash Cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0102] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, servers, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and memory) containing computer-usable program code.

[0103] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), servers, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0107] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0108] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An ASCII log parsing system based on a multi-mode CAN bus protocol, characterized in that, include: The file acquisition module is used to acquire ASC log files; The parsing module includes a regular expression list creation submodule, a validation criterion creation submodule, and a standardization submodule. The regular expression list creation submodule is used to parse the format of the ASC log file, the validation criterion creation submodule is used to validate the ASC log file, and the standardization submodule is used to perform standardization field transformation on the ASC log file and generate a parsing result list. The storage module includes a database submodule and a batch insertion and transaction management submodule. The database submodule is used to configure the data storage environment and generate the database. The batch insertion and transaction management submodule is used to batch insert the data of the parsed result list into the database and perform transaction management.

2. The system according to claim 1, characterized in that: The regular expression list building submodule matches the format of the ASC log file using a regular expression list. The regular expression list includes regular expressions that are mapped one-to-one with the format of the ASC log file. When performing the matching, each valid line of the ASC log file is processed, and the regular expressions are matched in a loop according to their order. The order is based on the frequency of the format in the actual scenario. After a successful match, the core fields are extracted according to the matched format, and the matching and verification of the subsequent formats are terminated.

3. The system according to claim 1, characterized in that: The verification criteria establishment submodule verifies ASC log files by establishing verification criteria, which include existence verification criteria, permission verification criteria, encoding verification criteria, and ultra-large file adaptation criteria. And / or, The standardization submodule includes: A filtering unit is used to filter invalid lines, including blank lines, comment lines, and lines with malformed formatting. The correction unit is used to remove leading and trailing whitespace, replace full-width spaces, unify capitalization, and unify separators.

4. The system according to claim 1, characterized in that, The database submodule includes: The database connection configuration unit is used to support multi-threaded shared connections, enable pre-write logging mode, support read and write concurrency, reduce synchronization level, set memory mapping, and set caching. The table structure creation unit is used to create a raw data table and a statistical data table. The raw data table is used to store the complete data of each parsed item in the parsing result list; the statistical data table is used to store the aggregated statistical information of each CAN_ID. The index optimization sub-unit is used to create composite indexes, which filter data based on specific channels and directions for high-frequency query scenarios.

5. The system according to claim 1, characterized in that, The batch insertion and transaction management submodule includes: The batching logic unit is used to split the parsing result list, divide the parsing result list into several batches, and implement batch processing. The transaction execution unit is used to call placeholder functions, batch execution functions, transaction guarantee functions, and field mapping verification functions for each batch of data. The placeholder functions are used to replace dynamic values ​​in the SQL statement. The batch execution functions are used to insert data from the parsed result list into the database in batches. The transaction guarantee functions are used to put the batch execution functions into the transaction management scope. If the insertion is successful, the transaction is committed. If an exception occurs, the transaction is rolled back. The field mapping verification function is used to adjust the core fields in the parsed result list to be strictly consistent with the field order of the SQL insertion statement, and to convert them into a list of tuples through list comprehensions.

6. The system according to claim 1, characterized in that, It also includes a query module, which includes: The condition generation unit is used to generate cached items and unique cache keys for query conditions; The linked query unit is used to verify the validity of the cache, including key existence verification and expiration time verification. The key existence verification is used to check whether the cache key exists in the preset cache dictionary. If the key does not exist, it is determined as a cache miss. If the key exists, the interval between the current time and the creation time is calculated by the expiration time verification. If it is greater than the cache expiration time, the cache item is determined to be expired, the cache item is deleted and determined as a cache miss. If it is less than or equal to the cache expiration time, the cache item is determined to be valid and the query result is obtained.

7. The system according to claim 1, characterized in that: When determining that a cached item is valid, obtaining the query result also includes: Update the access records, which include the number of accesses and the access time. If the list length exceeds 2, delete the oldest access record. Return the cached items for the query criteria; and / or, The process of determining that a cached item is expired, deleting the cached item, and determining a cache miss also includes: Query the database and filter query conditions by dynamically concatenating SQL statements; Obtain the field names of the filtered query conditions, and convert the list of tuples returned by the query into a dictionary list, ensuring that the format is consistent with the list of parsed results; If the query result of the query condition is not empty, write the query result of the query condition to the cache and trigger the cache eviction logic at the same time. Returns the query results based on the database query conditions.

8. A method for parsing ASCII logs based on a multi-mode CAN bus protocol, characterized in that, include: Get the ASC log file; Parse and validate the format of the ASC log file, perform standardized field transformation on the ASC log file, and generate a list of parsing results; Configure the data storage environment and generate the database; The data from the parsed result list is inserted into the database in batches, and transaction management is performed. The data increment of the parsed result list is also counted. Generate cached items and unique cache keys for the query conditions, and execute the query task.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program includes executable instructions that, when executed by a processor, implement the method of claim 8.

10. An electronic device, characterized in that, include: One or more processors; A memory for storing executable instructions of the processor, which, when executed by the one or more processors, cause the one or more processors to implement the method of claim 8.

Images