New energy automobile data dynamic processing and data quality monitoring system and method

Abstract

The invention provides a new energy automobile data dynamic processing and data quality monitoring system and method, and the system comprises a metadata service module which is used for storing an automobile identification code, basic information containing corresponding brands, automobile series and automobile types, and metadata information containing attribute fields of the brands, the automobile series and the automobile types; the data real-time processing module is used for performing flattening processing on the enriched multi-level burying point data, comparing the multi-level burying point data with the metadata information of the attribute fields, identifying abnormal conditions and dynamically updating the metadata information of the unidentified attribute fields; and the data alarm module is used for generating alarm information aiming at the abnormal condition identified by the data real-time processing module and sending a notice. According to the method, the data processing efficiency is greatly improved, and the data query cost, the exception positioning capability and the troubleshooting efficiency are reduced.

Description

Technical Field

[0001] This invention belongs to the field of new energy vehicle technology, specifically relating to a system and method for dynamic data processing and data quality monitoring of new energy vehicles. Background Technology

[0002] With the rapid development of the automotive industry, especially the continuous advancement of new energy vehicle technology, the amount of real-time data generated by new energy vehicles during operation is increasing rapidly, and the frequency of data updates is also constantly increasing. This places extremely high demands on real-time data processing. Existing data processing solutions typically employ traditional batch processing methods, which cannot meet the high-frequency, high-concurrency data reporting needs of new energy vehicles, resulting in problems such as insufficient real-time processing performance and severe processing delays.

[0003] In the prior art, CN118035060A discloses a method for dynamically generating a standard basic data model for event tracking. This method involves creating a tracking event configuration table, extracting and maintaining attribute data for tracking events, and dynamically generating the tracking data model using Spark tasks. While this method improves the development efficiency of tracking models and can respond to general business needs, it still has the following problems: 1. This solution uses a static field management mechanism, which cannot achieve real-time dynamic identification and processing of newly added attribute fields; 2. The proposed solution is a general-purpose data processing model for embedded data points, which is not optimized for the multi-level heterogeneous data characteristics of new energy vehicle data and cannot meet the special data processing needs of the new energy vehicle field. 3. It only performs basic statistics and monitoring on abnormal data, and fails to link with the alarm mechanism, so it cannot promptly notify relevant personnel to repair or investigate data anomalies; 4. There is a lack of effective methods for comprehensively assessing the overall data quality, making it difficult to accurately and timely perceive the overall quality status of various types of data.

[0004] Therefore, the current real-time data processing in the new energy vehicle field faces a series of problems, such as difficulty in processing multi-level, large-volume, heterogeneous data in real time, serious lag in data anomaly detection, low data query efficiency and high resource consumption, and inability to effectively assess overall data quality. There is an urgent need for a more efficient, real-time, and accurate dynamic data processing and quality monitoring solution to meet the pressing needs of new energy vehicle data management and analysis. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the aforementioned background technology by providing a dynamic data processing and data quality monitoring system and method for new energy vehicles. Through dynamic metadata management capabilities, it supports the real-time storage of newly added fields from sensor iterations, ensuring query performance under high-concurrency writes. It can promptly detect data anomalies based on streaming analysis and push anomaly alarms. It can also output overall data quality and automatically process multi-level and multi-structured data points, significantly improving data processing efficiency, reducing data query costs, enhancing anomaly location capabilities, and improving fault diagnosis efficiency.

[0006] The technical solution adopted in this invention is: a dynamic data processing and data quality monitoring system for new energy vehicles, comprising: The metadata service module is used to store vehicle identification codes and basic information including the corresponding brand, series, and model, as well as metadata information including attribute fields for each brand, series, and model. Real-time data processing module, used for Receive the embedded data stream reported by the vehicle in real time and parse the vehicle identification code in it; The metadata service module is invoked to obtain and supplement the vehicle's basic information into the embedded data record based on the vehicle identification code. The enriched multi-level tracking data is flattened and compared with the metadata information of the attribute fields to identify anomalies and dynamically update the metadata information of unidentified attribute fields. The data alarm module is used to generate alarm information and send notifications for abnormal situations identified by the real-time data processing module.

[0007] The above technical solution also includes a data query module, which is used to perform queries on the stored data based on the attribute field information provided by the metadata service module, and only query the registered attribute fields corresponding to the specified brand, car series or model, so as to reduce irrelevant data processing and improve query efficiency.

[0008] The above technical solution also includes a data quality assessment module, which is used to assess the quality of data from different brands, car series, and models, calculate the quality scores of attribute field dimensions and the overall quality score of the car series, and output a data quality assessment report.

[0009] In the above technical solution, the real-time data processing module includes a parsing task based on the Apache Flink streaming computing framework, which is used to traverse the multi-level nested structure of the vehicle-reported data points and flatten it according to the hierarchical key path to generate corresponding attribute field key-value pairs; and the real-time data processing module preloads and caches the metadata information of the attribute fields corresponding to each brand, vehicle series, and model into memory for dynamic comparison during real-time parsing; wherein, the basic information supplemented to the data point records is regarded as a system reserved field during the comparison process and does not participate in the validity verification of the data point fields.

[0010] In the above technical solution, if the real-time data processing module detects an undefined attribute field in the metadata during the comparison of the attribute fields with the metadata information, it automatically adds the attribute field to the attribute field information of the metadata service module as a new field, registers it as a new field with a preset default data type and no valid value range, and adds a corresponding field column to the underlying data storage table; at the same time, it generates a new field alarm information for the newly added attribute field and sends the alarm information to the data alarm module.

[0011] In the above technical solution, when the real-time data processing module compares the actual data of the attribute field with the metadata information, if it detects that the data type of the attribute field does not match the type defined in the metadata, it generates a type conversion exception alarm; if it detects that the value of the attribute field exceeds the valid value range defined in the metadata, it generates a value range exception alarm; the real-time data processing module sends the generated exception alarm information to the data alarm module to notify the relevant person in charge to handle it in real time.

[0012] In the above technical solution, the data quality assessment module periodically performs data quality assessment on the stored data, including: calculating a single-field quality score based on data integrity and data validity for each attribute field, and calculating the overall data quality score of the vehicle series based on the quality scores of each attribute field and their predetermined weights; the data quality assessment module outputs a data quality assessment report for each vehicle series, and lists the attribute fields with lower quality in order of their attribute field quality scores from low to high.

[0013] This invention also provides a method for dynamic processing and data quality monitoring of new energy vehicle data, comprising the following steps: Establish metadata storage to save vehicle identification numbers and corresponding brand, series, and model information, as well as attribute field metadata information for each brand, series, and model. The system acquires the embedded data reported by vehicles in real time, parses out the vehicle identification number (VIN), queries the corresponding brand, series, and model based on the VIN, and adds the brand, series, and model information to the embedded data record. Based on the brand, car series and model information, obtain the corresponding attribute field metadata information, traverse and parse the multi-level nested structure of the embedded data, and flatten the field key names according to the hierarchical relationship to generate a set of attribute field key-value pairs. The attribute field key-value pairs are compared with the pre-acquired attribute field metadata information: if the data type of an attribute field does not match the metadata definition or cannot complete the type conversion, a corresponding type conversion exception alarm message is generated; if the value of the attribute field exceeds the valid value range specified by the metadata, a value range exception alarm message is generated; if an attribute field that does not exist in the metadata is found, the attribute field is dynamically added to the metadata and marked as a new field. Filter out abnormal data that cannot be converted to the correct type, and write the normal data after leveling up into the storage table of the data warehouse for subsequent querying and analysis. The generated abnormal alarm information is sent to the designated alarm receiving terminal to notify relevant personnel to handle the abnormality in real time.

[0014] The above technical solution also includes: periodically conducting quality assessments on the data of each brand, car series, and model stored in the data warehouse, specifically including calculating a single-field quality score based on data integrity and validity for each attribute field, calculating the overall data quality score of the corresponding car series based on the quality scores of each attribute field and preset weights, and outputting a corresponding data quality assessment report.

[0015] The above technical solution also includes: when receiving a data query request for a specified brand, car series, or model, obtaining a list of registered attribute fields corresponding to the brand, car series, or model, and performing data query processing only based on the list of attribute fields, so as to reduce the amount of data processing for irrelevant fields and improve query efficiency.

[0016] The above technical solution, after dynamically adding undefined attribute fields to metadata, also includes: adding corresponding new attribute field columns to the underlying storage table of the data warehouse, and scanning the stored historical data to determine the existence of the new attribute fields in the historical data and whether there are any anomalies in the data.

[0017] The present invention also provides a non-transitory computer-readable storage medium storing a computer-executable program, wherein when the program stored thereon is executed by a processor, the processor performs the functions / steps described in the above technical solution.

[0018] The present invention also provides a data processing device configured in a server, comprising: a processor, a memory and a program thereon, the program causing the device to perform the functions of the system modules described above.

[0019] The beneficial effects of this invention are as follows: By organically combining a metadata service module and a real-time data processing module, this invention achieves real-time reception, VIN parsing, basic information enrichment, and unified flattening of multi-level nested embedded data reported by vehicles. It compares and dynamically updates metadata, simultaneously meeting the needs of dynamic expansion and high-concurrency stable processing, significantly improving the flexibility and reliability of data processing. Even if new fields are added during sensor iterations, they can be entered into the database in real time without downtime or manual table modifications; it adapts to massive concurrent writes, ensuring system stability.

[0020] Furthermore, this invention uses a metadata-driven data query module to limit queries to only registered fields of specified brands / car series / models, reducing the scanning of irrelevant fields, significantly reducing resource consumption and response latency for large table queries, improving query efficiency, and meeting real-time analysis needs.

[0021] Furthermore, this invention introduces a data quality assessment module, which uses a method based on single-field integrity / validity and overall vehicle series quality scoring to regularly output quality reports, providing comprehensive data health metrics. This allows for the timely detection and location of low-quality fields and vehicle models, assisting in operation and maintenance and decision-making, and improving overall data reliability.

[0022] Furthermore, this invention utilizes the Apache Flink streaming framework's real-time parsing task to flatten multi-level JSON and preload field metadata into a HashMap cache. By leveraging memory caching and streaming computation, it achieves millisecond-level field parsing and validation, meeting the requirements of high-throughput, low-latency scenarios.

[0023] Furthermore, when encountering an unknown field, this invention automatically and dynamically registers the new field and updates the underlying table structure, while generating an alarm for the newly added field, achieving zero-downtime expansion of the data tracking model; it also promptly notifies the data team to supplement the field definition, avoiding data loss and reporting blind spots.

[0024] Furthermore, this invention performs type conversion and value range validation on defined fields, generating and pushing alerts for type conversion or value range anomalies in real time. It captures data format or value anomalies immediately, triggering repair or maintenance processes and shortening the troubleshooting cycle.

[0025] Furthermore, this invention calculates individual field quality scores for stored data based on field completeness and validity dimensions, and then weights and aggregates these scores into an overall quality score for the entire vehicle series. This quantifies the data quality of each vehicle model, supports cross-model comparisons, and provides a basis for data governance and improvement.

[0026] Furthermore, the method of this invention comprehensively covers the entire process of metadata management, real-time data reception, basic information enrichment, flattened parsing, anomaly detection, storage, and alarm push. It realizes an end-to-end integrated real-time processing solution, simplifies implementation complexity, and ensures consistency and reliability of the processing chain.

[0027] Furthermore, the method of the present invention incorporates a periodic data quality assessment step to achieve periodic quality monitoring. The assessment frequency can be adjusted as needed to continuously track the health status of the data and control the long-term data quality trend.

[0028] Furthermore, in the method of the present invention, a list of fields is dynamically obtained and the query range is clipped for each query request. This works in conjunction with the system-side query module to ensure that the query and real-time processing are equally efficient, and to avoid interference from irrelevant fields.

[0029] Furthermore, in the method of the present invention, after a new field is registered, the table structure is automatically updated and historical data is scanned to confirm the data quality of the new field. Metadata is supplemented and historical data can be backtracked to ensure that the new field is subject to quality monitoring from the moment it goes online.

[0030] Furthermore, this invention protects computer executable programs in the form of non-transitory storage media, achieving physical isolation coverage of any function of the system claims, expanding the scope of protection for software product forms, and effectively preventing others from evading responsibility by simply changing the carrier or copying the program.

[0031] Furthermore, the present invention protects system functions in the form of a device configured on a server, covering the processor, memory and the overall implementation of the program, ensuring that as long as the corresponding module is deployed on a specific hardware platform, it will fall within the scope of protection of the claims, thus blocking the implementation path of the system. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the real-time data processing flow of the present invention. Detailed Implementation

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but these descriptions do not constitute a limitation on the present invention.

[0034] Example 1 like Figure 1 As shown, the present invention provides a dynamic data processing and data quality monitoring system for new energy vehicles, comprising: The metadata service module is used to store vehicle identification codes and basic information including the corresponding brand, series, and model, as well as metadata information including attribute fields for each brand, series, and model. Real-time data processing module, used for Receive the embedded data stream reported by the vehicle in real time and parse the vehicle identification code in it; The metadata service module is invoked to obtain and supplement the vehicle's basic information into the embedded data record based on the vehicle identification code. The enriched multi-level tracking data is flattened and compared with the metadata information of the attribute fields to identify anomalies and dynamically update the metadata information of unidentified attribute fields. The data alarm module is used to generate alarm information and send notifications for abnormal situations identified by the real-time data processing module.

[0035] The above technical solution also includes: The data query module is used to perform queries on the stored data based on the attribute field information provided by the metadata service module, and only queries the registered attribute fields corresponding to the specified brand, car series or model, so as to reduce irrelevant data processing and improve query efficiency. The data quality assessment module is used to assess the quality of data from different brands, car series, and models, calculate the quality scores for attribute fields and the overall quality score for the car series, and output a data quality assessment report.

[0036] The various functional modules of the entire system are distributed across the vehicle and server sides, each undertaking different responsibilities: The vehicle-mounted terminal consists of the vehicle's own sensors and a T-Box. Various physical sensors are installed on the vehicle to collect data such as battery voltage, motor temperature, and vehicle speed. The T-Box acts as an on-board gateway, responsible for collecting data from the sensors / controllers and uploading the data via a wireless network. Located on the vehicle, the T-Box is the exit point for the data source.

[0037] Cloud / Server Side: This includes all subsequent real-time data processing modules, metadata service modules, data query modules, data quality assessment modules, and data alarm modules, and is typically deployed on the manufacturer's cloud platform or data center.

[0038] On the one hand, the real-time data processing module has data access and processing components: the Kafka cluster is responsible for data receiving and buffering, and the Flink computing cluster is responsible for real-time streaming computing processing.

[0039] On the other hand, the metadata service module runs as an independent service in the cloud, used for centralized storage and providing metadata queries and updates. The cleaned data is stored in a cloud-based data warehouse or database cluster for business queries and analysis. This data warehouse may be deployed in a data center by a big data platform (such as Hive / ClickHouse).

[0040] Finally, the data alarm module and the quality assessment module also run in the cloud to utilize centrally stored data and high-performance computing resources to complete alarm push and quality calculation.

[0041] In general, apart from the sensors and T-Box, the other functional modules (Kafka, Flink, metadata service, data warehouse, alarm handling, quality assessment, query interface, etc.) are all deployed on the cloud or data center servers and managed uniformly by the backend system.

[0042] Specifically, various sensors on new energy vehicles generate vehicle status data in real time. This data is first collected by the onboard T-Box (vehicle communication box) for processing and transmission. The T-Box packages the sensor data into a specific format (such as JSON messages) and uploads the data to the cloud entry point via a wireless network.

[0043] In the cloud, the data streams reported by these vehicles are first received by the Kafka distributed message queue. Kafka acts as a data buffer layer in the system, and the data for each vehicle (distinguished by the Vehicle Identification Number, VIN) is written to a specific Topic partition in Kafka. Through Kafka's publish-subscribe mechanism, the vehicle T-Box (data producer) and the backend stream processing cluster (data consumer) are decoupled. That is, the vehicle data first enters the raw data Topic in Kafka (such as vehicle_raw_data) and is temporarily stored as raw sensor messages.

[0044] Once vehicle data enters Kafka, it is consumed and processed by the backend real-time data processing module. This module uses the Apache Flink stream processing engine and is deployed on a cloud cluster. The real-time processing flow is as follows: When a Flink job starts, it loads and caches all vehicle attribute field definitions from the metadata service. Then, Flink continuously reads the latest vehicle data messages from the Kafka raw data topic, parsing and processing each message individually. Each uploaded vehicle data message is typically a multi-level JSON structure, potentially containing nested key-value pairs or arrays. Flink first extracts the Vehicle Identification Number (VIN) from the message, then uses the VIN to query the metadata service for basic information such as the vehicle's brand, model, and type, adding this information to the data.

[0045] Next, based on the vehicle's brand / series / model identifier, the corresponding model's defined attribute field list and its type, valid value range, and other metadata are retrieved from the previously cached HashMap.

[0046] Subsequently, Flink traverses, parses, and flattens the JSON data. The specific rule is: recursively traverse the entire hierarchical structure of the JSON, expanding nested objects and arrays into single-level key-value pairs. Key-value pairs with values ​​of basic types (non-object, non-array) are directly retained; if the value is an object or array, its internal fields are extracted, and the parent key name and the separator "#" are added before the key name to form a flattened composite key. For example, the original JSON {"a": {"b": 1}, "c":[2, {"d": 3}]} becomes {"a#b": 1, "c#0": 2, "c#1#d": 3} after flattening. In this way, all multi-level data is uniformly flattened into attribute fields in the form of "parent key #child key", facilitating subsequent storage and retrieval.

[0047] While flattening the data, Flink uses cached metadata to validate and categorize the values ​​of each field: Known field type checks: If a flattened field is registered in the metadata, its value is attempted to be converted to the data type specified by the metadata. For example, a type conversion error will occur if a field is expected to be an integer but is received as a string. If such a type conversion exception occurs, the field is not saved, and an alert event is written to the Kafka "Alerts" topic. Similarly, if a field exists but its value exceeds a predefined valid range, an alert event will also be generated. In this case, the field value will still be saved, but the out-of-range condition will be marked for alert notification.

[0048] Handling Unknown New Fields: If a field is encountered that does not exist in the current vehicle model's metadata, it is considered a newly added data tracking field. The system will dynamically expand the metadata: immediately register the new field in the metadata service (initially set to string by default, with no preset valid value range). Simultaneously, by executing DDL statements in the data warehouse table, a corresponding column will be added to the underlying storage to accommodate the data from this new field. For such unknown fields, a new field alert event will also be generated and written to the alert topic to notify relevant personnel of the new data field's appearance.

[0049] Preferably, the dynamic DDL automatic generation process is as follows: 1. Triggering conditions When the metadata service detects an unregistered field name in an upstream message, it triggers the DDL generation process.

[0050] For example: If the JSON key battery#tempCell5 is received, there is no record for that in the field definition table.

[0051] 2. Extraction of field metadata Read field names, data types (string / number / boolean), nullability, and default value strategies from the reported JSON.

[0052] 3. DDL statement template assembly Use parameterized templates to generate ALTER TABLE statements.

[0053] 4. Script execution and idempotency verification The above DDL is executed via JDBC / Connector in a Flink job or initialization script.

[0054] Before execution, query the table metadata to confirm that the column does not exist, to avoid repeated errors: 5. Metadata Registration After the DDL is executed successfully, the metadata information of the new field is written to the metadata_fields table for use by the subsequent validation and query modules.

[0055] 6. Exception handling and rollback If DDL execution fails, an error log is recorded and an alarm module is triggered, requiring manual intervention for confirmation. Failed transactions are rolled back to ensure data consistency.

[0056] After completing the above verification, Flink filters out abnormal fields that cannot be converted (i.e., data with incorrect types does not enter the warehouse). Then, the cleaned and flattened complete data records are written to the detail table in the data warehouse for storage. The data warehouse is typically deployed in a cloud-based data lake or distributed storage (such as a Hive data warehouse or large data tables). Each record contains basic vehicle information and all attribute fields reported in that instance. Through a mechanism of dynamically adding new columns, the schema of the data warehouse table can automatically expand as sensor fields iterate. At this point, the process of transmitting a single piece of vehicle-reported data from the vehicle to the cloud, undergoing real-time processing, and finally storing it in the database is complete.

[0057] Meanwhile, any anomalies (type errors, out-of-bounds values, new fields) discovered during the data processing have generated corresponding alert events and are temporarily stored in the Kafka alert topic awaiting processing.

[0058] Specifically, the metadata service module centrally manages all key metadata related to the system. Its stored content mainly includes two parts: Vehicle Basic Information Table: Stores static information for all connected vehicles, such as the Vehicle Identification Number (VIN) mapped to its brand, model series, vehicle type, powertrain type (pure electric BEV / range-extended REV), standard range, manufacturing date, and registration date. This mapping allows for quick identification of the vehicle's category based on the VIN, facilitating subsequent data processing and statistics.

[0059] The data entry field definition table stores metadata about all sensor / data entry field definitions in the system. Each definition includes the brand, vehicle series, vehicle model, field name, parent module (e.g., motor, vehicle infotainment system), parent field (if nested), data type (integer, floating point, character, etc.), and valid value range. This is essentially a "data dictionary" describing the data format for different vehicle models and the meaning and constraints of each field.

[0060] The metadata service module provides query and update interfaces for use by other modules. In real-time Flink jobs, metadata is fully utilized: when a Flink task starts, it pulls a list of field definitions for all car models at once through the service interface, and caches the field metadata for each brand-series-model combination in an in-memory HashMap. The cache uses a key-value pair structure, where the key is a combination of strings "brand#series#model" and the value is a set of all attribute field information for that car model. With the help of the O(1) lookup feature of HashMap, stream processing can quickly retrieve the field specifications for the corresponding car model when each piece of data arrives.

[0061] Preferably, the basic information of the metadata service interface is shown in Table 1: Table 1 Basic Information of Metadata Service Interface The parameters of the metadata service interface are shown in Table 2: Table 2 Metadata Service Interface Parameters The response results of the metadata service are shown in Table 3: Table 3 Explanation of the response results of the metadata service A typical example of a metadata service request is as follows: GET / api / metadata / vin / query?vin=Lxxxxx12345; This uses the HTTPGET method to pass the VIN parameter as a URL query string, which conforms to the RESTful specification.

[0062] A typical response example for metadata services is as follows: Successful response (HTTP 200) { "code": 200, "message": "success", "data": { "vin": "Lxxx12345", "brand_name": "xx brand", "series_name": "xxc car series", "type_name": "xx car model" }} The code / message is responsible for general status returns, while the data object carries business fields. The data contains the VIN itself and its corresponding brand, series, and model information, which can be directly used for subsequent stream processing enrichment.

[0063] When processing each piece of data, Flink uses the VIN to call the metadata service to retrieve the vehicle's brand / model information for data enrichment. Subsequently, without needing to access the service again, it can directly retrieve the field types and threshold definitions for that vehicle model from the local HashMap to validate the data. HashMap caching significantly improves metadata lookup efficiency in high-concurrency scenarios, avoiding frequent remote calls to the metadata service.

[0064] The metadata service module continuously runs as the authoritative data source, accepting update requests. For example, when a new field is detected, the Flink processor registers the field's metadata information (name, default type, no initial range) with the metadata service via an interface and adds a corresponding column to the underlying data table. Subsequently, after confirming the meaning of the new field, relevant personnel can refine the field's type and valid value range definition in the metadata service. After the metadata service is updated, the new metadata will be used for the next Flink task startup or periodically refreshed cache, thereby dynamically expanding the system's data model.

[0065] Therefore, the relationship between HashMap caching and metadata services is as follows: the former improves query speed for real-time verification, while the latter provides persistent storage to ensure metadata consistency and update sources. Together, they enable high-speed access and dynamic maintenance of metadata.

[0066] Specifically, anomaly detection logic is embedded in the real-time data processing module. Once a data anomaly is detected, an alarm mechanism is immediately triggered, which then coordinates with relevant personnel or systems for handling. The specific logic for triggering the alarm is as follows: Type conversion error: When the actual type of the uploaded data does not match the type defined in the metadata (e.g., a field that should be a number is received as a non-numeric string), Flink cannot complete the type conversion and determines that the data in that field is invalid. In response, the system discards the field value and does not write it to the database, and immediately generates an alert message written to the Kafka alerts topic. The alert content includes information such as the vehicle identifier, field name, and error type.

[0067] Value range out of bounds exception: If a field value is successfully parsed but exceeds the defined normal value range (e.g., voltage exceeds a reasonable range), the data is considered potentially abnormal but still acceptable for storage. Flink will generate a value range exception alarm message and add it to the alarm topic when storing the data in the database.

[0068] Adding an unknown field: When an undefined field (i.e., a field name not present in the metadata) is detected in the data, the system will dynamically register this new field and add it as a storage column. Then, it will issue a new field alert message for this field. This alert is used to notify the data team of the presence of an unregistered new data point for subsequent confirmation and processing.

[0069] All these alert events are first written to the alerts topic in Kafka as asynchronous messages. Subsequently, the data alerting module consumes these alert messages and pushes notifications.

[0070] The data alerting module is typically a Flink stream processing task or a standalone service, deployed in the cloud and subscribed to alerts topics. Upon startup, it loads the alert configurations for each brand / vehicle series / model, such as the responsible parties and notification methods (SMS, email, or app push notifications) for different alert types. When a new alert message arrives, the alerting module looks up the pre-configured list of responsible parties and their contact information based on the vehicle brand / model, and immediately sends the alert information to the relevant person or system. Different types of alerts will have different processing and linkage flows: For data anomalies such as type conversion errors or out-of-range value domain warnings, operation and maintenance / R&D personnel can understand from the notification which vehicle has what kind of data anomaly. If it is determined that the cause of the anomaly is a problem with the on-vehicle data collection or reporting logic (for example, a software bug in the T-Box causes incorrect data formats), relevant technical personnel will start to repair the program vulnerability and remotely upgrade the on-vehicle software via OTA (Over-the-Air) to correct the data reporting format in the next version. If the anomaly is due to the vehicle hardware itself (sensor failure or abnormal fluctuations), the system will forward the warning message to the vehicle user or after-sales service, who will notify the vehicle owner to repair the vehicle in a timely manner. This mechanism enables data anomalies, which may have been discovered long after the fact, to be perceived in real time, responded to promptly, and relevant parties to be linked up for processing immediately, significantly improving the efficiency of anomaly perception and handling.

[0071] For new data field warnings, data managers will analyze the received warnings: First, confirm whether the field is an expected addition (for example, new sensor data is added when a new function is launched). If it is confirmed that it is a valid field added normally, it is necessary to complete the meta-information configuration of this field in the metadata service (such as the correct data type and reasonable value domain range), and save the update. After the metadata update is completed, the system will automatically trigger a task to scan and verify the historical data that has been stored in the database to check whether there are abnormal values or situations that need to be supplemented and processed for the data that entered during the period when the new field was not defined. If problems are found, further remedies can be taken to ensure data consistency. If it is verified that the field is not a real new data point (that is, data that should not appear, such as a pseudo-field caused by an incorrect reporting format), it is necessary to investigate the cause and may clean or delete the relevant data, and at the same time do not add this field to the metadata to maintain the model's correctness. All warning events and subsequent processing results will also be recorded and archived to facilitate the statistics of the frequency and type of anomalies for each brand / model / vehicle type, providing a basis for quality improvement.

[0072] Through the above warning mechanism, the system closely combines data anomaly detection and notification, achieving linkage with users and the background operation and maintenance system: It can not only notify users to handle problems with the vehicle itself as soon as possible, but also notify development and operation and maintenance personnel to perform system-level interventions such as OTA updates for problems with the data collection system. This multi-party linkage ensures that data problems are perceived and resolved in a timely manner, guaranteeing the overall data quality.

[0073] Specifically, the data quality assessment module is mainly responsible for regularly detecting and scoring the data of each brand / vehicle series / model, and providing a quantitative report on the overall data health. According to the description in the technical disclosure document, this module runs according to a set cycle (such as daily reports or once a week), is triggered regularly, and does not rely on real-time warning events to start. That is to say, it is more like a scheduled task or batch processing job that scans the accumulated data at fixed times and evaluates quality indicators.

[0074] In terms of operation, data quality assessment can be performed by an offline batch processing engine (such as Spark / Flink batch mode) to conduct statistical analysis on historical data in the data warehouse. The assessment includes two levels: single-field quality and overall vehicle series quality. First, a single-field quality score is calculated for each data field of each vehicle model. The quality score considers two dimensions: completeness (data missing cases) and validity (data values ​​within a valid range). Specifically, for a given field, the total number of reports, the number of non-empty valid values, and the number of times they fall within a predefined normal range are counted within the assessment period. Then, the quality score for that field is calculated according to a pre-set weighted formula. For example, completeness can be defined as the percentage of non-empty values, and validity as the percentage of valid values; different dimensions are weighted to obtain a field score from 0 to 100.

[0075] Preferably, the following formula is used to score the quality of a single field: in and As a weighting factor for completeness and effectiveness, and, , This represents the number of valid (non-empty) values ​​in this field. The number of this field within its valid range. This represents the total number of items in this field.

[0076] Next, summarize the quality scores of all fields under this vehicle model to calculate the overall data quality score for the entire vehicle series. A common practice is to assign different weights to each field's score according to its importance, and then sum them to obtain the overall score, using the following formula: in Assign weights to each field. The quality assessment score is given for each field.

[0077] Preferably, the recommended initial values ​​and adjustable ranges of each weight are shown in Table 4 (those skilled in the art can fine-tune them within a given range based on actual business scenarios and historical operating data to balance completeness, accuracy, real-time performance, and consistency): Table 4 Recommended initial values ​​and adjustable ranges for each weight The importance weights for each field can be set and normalized as shown in Table 5: Table 5 Importance Weights of Each Field The recommended values ​​mentioned above are all derived from the empirical results obtained through large-scale offline backtesting and online simulation operation on different vehicle models and operating conditions after the present invention was launched on a real vehicle data processing platform. Those skilled in the art can make appropriate adjustments within the given range in the table according to the specific vehicle brand, reporting frequency, and business priorities, without any additional creative work, to fully reproduce the quality scoring function of the present invention.

[0078] The evaluation results generate a report, including an overall score for each brand / model / vehicle series, and a list of fields ranked from lowest to highest quality score. By reviewing the top fields with the worst quality, the data team can quickly pinpoint which sensor data has significant issues (such as frequent missing or out-of-range data). This information helps to make targeted improvements to data collection (such as replacing faulty sensors or optimizing reporting frequency) or refine metadata definitions.

[0079] The data quality assessment module typically runs automatically at a fixed frequency without manual intervention. However, additional assessments can be manually triggered in special circumstances. For example, after new field metadata is added, the system automatically scans the historical data for that field, which can be considered a specific quality check. Overall, routine data quality assessments are scheduled periodically to ensure continuous monitoring of data quality trends across different vehicle models, rather than being directly triggered by real-time alarms. Through periodic quality assessment reports, managers can gain a macro-level understanding of the data reliability for different vehicle models and promptly identify long-standing data quality vulnerabilities.

[0080] Specifically, the data query module consists of two main sub-modules: the attribute field metadata query sub-module and the embedded data query sub-module.

[0081] The attribute field metadata query submodule provides metadata summaries by brand, car series, and model level, including: the total number of registered attribute fields at that level and the amount of data reported for each attribute field.

[0082] The attribute field metadata query submodule receives query requests, obtains the brand / car series / model identifier specified in the request; calls the metadata service interface to pull the list of attribute fields at this level and their data table mapping relationship; performs statistics based on the data warehouse (e.g., Hive / ClickHouse); and returns the statistical results to the caller by comparing the field name with the reported quantity.

[0083] All metadata is managed uniformly through a metadata service, ensuring consistency between field definitions and downstream queries. Partitioned tables or columnar storage are used, with tables or partitions created by brand, vehicle series, or model to improve statistical efficiency. Pagination and sorting (e.g., descending order by reported volume) are supported, facilitating quick location of missing or rarely used fields.

[0084] The event tracking data query submodule returns only the event tracking data corresponding to the registered attribute fields based on the brand / car series / model, avoiding scanning all columns at once and significantly improving query performance.

[0085] The event tracking data query submodule receives query requests, parses out brand, vehicle series, model, and optional filter conditions (such as time range, specific vehicle VIN list, etc.); calls the metadata service to obtain the corresponding attribute field list; and constructs the projection push-down SQL. <field1> , <field2>…; dynamically assemble lists returned by metadata services; submit SQL to the underlying database (such as Presto, ClickHouse) for execution and return the results in JSON or tabular form; support streaming pagination for large result sets to reduce the pressure of single network transmission.

[0086] The data tracking submodule selects only registered fields, avoiding invalid field scanning and saving I / O and computing resources. At the query gateway layer, a short-term cache (TTL) can be set for hot queries to avoid repeated calls to the metadata service. Combined with the authentication module, different user roles can be granted different brand / vehicle data access ranges. Both synchronous and asynchronous query modes are supported to meet the needs of real-time analysis and batch export of historical data.

[0087] Through the cooperation of the above two-level sub-modules, the data query module can: quickly assess the coverage of tracking points for each vehicle model, assist in the supplementation of tracking points and model maintenance; efficiently retrieve the required tracking point data, reduce query costs, and meet the real-time or offline analysis needs in multiple scenarios.

[0088] Example 2 like Figure 2 As shown, the present invention also provides a method for dynamic processing and data quality monitoring of new energy vehicle data, comprising the following steps: Establish metadata storage to save vehicle identification numbers and corresponding brand, series, and model information, as well as attribute field metadata information for each brand, series, and model. The system acquires the embedded data reported by vehicles in real time, parses out the vehicle identification number (VIN), queries the corresponding brand, series, and model based on the VIN, and adds the brand, series, and model information to the embedded data record. Based on the brand, car series and model information, obtain the corresponding attribute field metadata information, traverse and parse the multi-level nested structure of the embedded data, and flatten the field key names according to the hierarchical relationship to generate a set of attribute field key-value pairs. The attribute field key-value pairs are compared with the pre-acquired attribute field metadata information: if the data type of an attribute field does not match the metadata definition or cannot complete the type conversion, a corresponding type conversion exception alarm message is generated; if the value of the attribute field exceeds the valid value range specified by the metadata, a value range exception alarm message is generated; if an attribute field that does not exist in the metadata is found, the attribute field is dynamically added to the metadata and marked as a new field. Filter out abnormal data that cannot be converted to the correct type, and write the normal data after leveling up into the storage table of the data warehouse for subsequent querying and analysis. The generated abnormal alarm information is sent to the designated alarm receiving terminal to notify relevant personnel to handle the abnormality in real time.

[0089] Specifically, the entire process of the method of the present invention, from vehicle data generation to quality assessment, is as follows in chronological order: 1. Onboard data generation and uploading: Various sensors on new energy vehicles send signals to the onboard T-Box. The T-Box preprocesses the data, packages it into formats such as JSON, and uploads it to the cloud in real time via wireless network.

[0090] 2. Cloud Data Reception (Kafka): Vehicle data is first buffered in the raw data topic of the Kafka message queue in the cloud. Data streams from different vehicles are allocated to different partitions in Kafka according to their VINs to ensure the sequential and parallel processing of subsequent data. Kafka decouples the vehicle-side and cloud-side consumers, ensuring that vehicle uploads are not affected by the backend processing speed.

[0091] 3. Real-time processing initialization (Flink): After the cloud-based Flink stream processing job starts, it immediately loads the field definition information of all vehicle models from the metadata service and builds a HashMap cache for fast querying. Simultaneously, it prepares to subscribe to the vehicle data topic from Kafka and enters a continuous consumption state.

[0092] 4. Single data processing flow: Whenever a new vehicle data message arrives in Kafka, Flink consumes the JSON data and performs a series of steps: VIN Recognition and Enrichment: Extract the vehicle's VIN code and query the metadata service to obtain information such as the vehicle's brand, series, and model, then add it to the current data record. The metadata service pre-registers basic information such as brand, series, and model as reserved fields with string type and unlimited valid value range.

[0093] Field retrieval specifications: Based on the vehicle model, retrieve all field types and thresholds defined for that model from the in-memory HashMap for subsequent validation.

[0094] JSON Expand: Traverses the JSON structure and expands nested field names into a flat collection of key-value pairs in the form of "level 1#level 2".

[0095] Data validation and alert generation: For each flattened field, it is compared with the metadata: If the field is defined, its type and value range are checked; if the type does not match, it is discarded and a type exception alert is recorded; if the value is out of range, a value range exception alert is recorded. If the field is not defined, a new field (default type string) is dynamically registered and added as a storage column, while an alert for the newly added field is recorded. These alert messages are asynchronously written to the Kafka alert topic, awaiting notification.

[0096] Data Ingestion: Cleaned, normal data (excluding incorrectly typed entries) is assembled into complete records and written to a detailed table in the cloud data warehouse. The data in the warehouse is categorized by vehicle model or presented in a unified large table (containing all possible fields), available for subsequent querying and analysis.

[0097] 5. Alarm Notification Processing: Messages in the alarm topic are consumed in real time by the alarm processing module. Based on the vehicle and anomaly type in the alarm content, the system searches for the preset notification strategy and pushes the anomaly details to the relevant responsible person or directly notifies the vehicle owner. For example, a type / value range anomaly will notify the backend R&D or operations personnel, guiding them to repair the software via OTA or contact the user to inspect the vehicle; a new field alarm will notify the data management personnel to confirm and improve the metadata definition of the new field. All alarms are recorded and archived for statistical analysis of anomaly distribution.

[0098] 6. Data Query and Utilization: The system provides data query functionality, allowing maintenance or business personnel to retrieve relevant data by brand, vehicle series, or model. On one hand, metadata (such as the list of fields and data volume reported for a specific vehicle model) can be queried to quickly identify any unexpected or missing tracking points. On the other hand, detailed vehicle-reported data can be queried. In this case, the metadata service provides a list of valid fields for that vehicle model, allowing the query to select only these columns, thus reducing the scanning of irrelevant columns and improving query performance. Metadata-driven queries avoid slow queries and resource waste caused by inconsistencies in fields across different vehicle models.

[0099] 7. Regular Data Quality Assessment: Following a pre-defined schedule (e.g., daily at midnight), the quality assessment module runs to analyze the quality of recent data for each vehicle model in the data warehouse. It calculates the completeness and validity indicators for each field and generates a quality score, which is then weighted and summarized to form the overall data quality score for each vehicle series. The assessment results generate a report listing fields and vehicle models with poor data quality for continuous improvement. This process runs independently in the background and does not affect real-time data processing.

[0100] Through the above process overview, the end-to-end working mechanism of this invention, from data acquisition, real-time processing, and anomaly alerting to subsequent storage, retrieval, and quality assessment, is clearly visible. The entire system achieves real-time access to vehicle-side data and efficient cloud processing, utilizing dynamic metadata management to adapt to constantly changing data structures. Simultaneously, combined with streaming anomaly detection and alarm notification mechanisms, data issues are promptly reported to relevant users or systems for processing. Finally, through regular data quality assessments, the reliability of data for each vehicle model is comprehensively measured, providing a basis for improvement. This series of modules operates sequentially, constituting a complete dynamic data processing and quality monitoring process for new energy vehicles.

[0101] Example 3 The present invention also provides a non-transitory computer-readable storage medium storing a computer-executable program, wherein when the program stored thereon is executed by a processor, the processor performs the functions / steps described in the above technical solution.

[0102] Example 4 The present invention also provides a data processing device configured in a server, comprising: a processor, a memory and a program thereon, the program causing the device to perform the functions of the system modules described above.

[0103] The key terms in this embodiment are defined as follows: Flink: Apache Flink is a distributed streaming data processing framework that supports high-throughput, low-latency real-time computing.

[0104] HashMap: A hash table data structure that uses key-value pairs for storage to achieve O(1) time complexity for queries.

[0105] Kafka is a distributed message queue system that uses a publish-subscribe model. In this embodiment, it serves as a data buffer layer, storing the raw data streams of different vehicle VINs through Topic partitioning, thereby decoupling the production end (vehicle-mounted T-Box) from the consumer end (Flink computing cluster).

[0106] Topic: A logical message classification unit in Kafka. For example: vehicle_raw_data: stores unprocessed raw sensor messages; alerts: stores abnormal events output by the streaming computing layer.

[0107] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

[0112] The contents not described in detail in this specification are existing technologies known to those skilled in the art. < / field1>

Claims

1. A system for dynamic data processing and data quality monitoring of new energy vehicles, characterized in that, include: The metadata service module is used to store vehicle identification codes and basic information including the corresponding brand, series, and model, as well as metadata information including attribute fields for each brand, series, and model. Real-time data processing module, used for Receive the embedded data stream reported by the vehicle in real time and parse the vehicle identification code in it; The metadata service module is invoked to obtain and supplement the vehicle's basic information into the embedded data record based on the vehicle identification code. The enriched multi-level tracking data is flattened and compared with the metadata information of the attribute fields to identify anomalies and dynamically update the metadata information of unidentified attribute fields. The data alarm module is used to generate alarm information and send notifications for abnormal situations identified by the real-time data processing module.

2. The system according to claim 1, characterized in that: It also includes a data query module, which is used to perform queries on the stored data based on the attribute field information provided by the metadata service module, and only query the registered attribute fields corresponding to the specified brand, series or model, so as to reduce irrelevant data processing and improve query efficiency.

3. The system according to claim 1, characterized in that: It also includes a data quality assessment module, which is used to assess the quality of data from different brands, car series, and models, calculate the quality scores of attribute fields and the overall quality score of the car series, and output a data quality assessment report.

4. The system according to claim 1, characterized in that, The real-time data processing module includes a parsing task based on the Apache Flink streaming computing framework. This task traverses the multi-level nested structure of the vehicle-reported data points and flattens it according to the hierarchical key path to generate corresponding attribute field key-value pairs. Furthermore, the real-time data processing module preloads and caches the metadata information of the attribute fields corresponding to each brand, vehicle series, and model into memory for dynamic comparison during real-time parsing. The basic information added to the data point records is considered a system-reserved field during the comparison process and does not participate in the validation of the data point field validity.

5. The system according to claim 1, characterized in that, During the comparison of the attribute fields with the metadata information, if the real-time data processing module detects an attribute field that is not defined in the metadata, it automatically adds the attribute field to the attribute field information of the metadata service module as a new field, registers it as a new field with a preset default data type and no valid value range, and adds a corresponding field column to the underlying data storage table; at the same time, it generates a new field alarm message for the newly added attribute field and sends the alarm message to the data alarm module.

6. The system according to claim 1, characterized in that, When comparing the actual data of the attribute field with the metadata information, the real-time data processing module generates a type conversion exception alarm if it detects that the data type of the attribute field does not match the type defined in the metadata; if it detects that the value of the attribute field exceeds the valid value range defined in the metadata, it generates a value range exception alarm. The real-time data processing module sends the generated exception alarm information to the data alarm module to notify the relevant person in charge to handle it in real time.

7. The system according to claim 3, characterized in that, The data quality assessment module periodically performs data quality assessments on the stored data, including: calculating a single-field quality score based on data integrity and data validity for each attribute field, and calculating the overall data quality score of the vehicle series based on the quality scores of each attribute field and their predetermined weights; the data quality assessment module outputs a data quality assessment report for each vehicle series, and lists the attribute fields with lower quality in order of their attribute field quality scores from low to high.

8. A method for dynamic data processing and data quality monitoring of new energy vehicles, characterized in that, Includes the following steps: Establish metadata storage to save vehicle identification numbers and corresponding brand, series, and model information, as well as attribute field metadata information for each brand, series, and model. The system acquires the embedded data reported by vehicles in real time, parses out the vehicle identification number (VIN), queries the corresponding brand, series, and model based on the VIN, and adds the brand, series, and model information to the embedded data record. Based on the brand, car series and model information, obtain the corresponding attribute field metadata information, traverse and parse the multi-level nested structure of the embedded data, and flatten the field key names according to the hierarchical relationship to generate a set of attribute field key-value pairs. The attribute field key-value pairs are compared with the pre-acquired attribute field metadata information: if the data type of an attribute field does not match the metadata definition or cannot complete the type conversion, a corresponding type conversion exception alarm message is generated. If the value of an attribute field exceeds the valid value range specified by the metadata, a value range exception alarm message will be generated; If an attribute field is found that does not exist in the metadata, the attribute field will be dynamically added to the metadata and marked as a new field. Filter out abnormal data that cannot be converted to the correct type, and write the normal data after leveling up into the storage table of the data warehouse for subsequent querying and analysis. The generated abnormal alarm information is sent to the designated alarm receiving terminal to notify relevant personnel to handle the abnormality in real time.

9. The method according to claim 8, characterized in that, Also includes: The data of each brand, car series, and model stored in the data warehouse is regularly evaluated for quality. Specifically, this includes calculating a single-field quality score based on data integrity and validity for each attribute field, calculating the overall data quality score of the corresponding car series based on the quality scores of each attribute field and preset weights, and outputting a corresponding data quality evaluation report.

10. The method according to claim 8, characterized in that, Also includes: When a data query request for a specified brand, car series, or model is received, a list of registered attribute fields corresponding to the brand, car series, or model is obtained, and data query processing is performed only based on the list of attribute fields to reduce the amount of data processing for irrelevant fields and improve query efficiency.

11. The method according to claim 8, characterized in that, After dynamically adding undefined attribute fields to the metadata, the process also includes: adding corresponding new attribute field columns to the underlying storage table of the data warehouse, and scanning the stored historical data to determine the existence of the new attribute fields in the historical data and whether there are any anomalies in the data.

12. A non-transitory computer-readable storage medium storing a computer-executable program, wherein the program stored thereon, when executed by a processor, causes the processor to perform the functions / steps described in any one of claims 1-7 or 8-11.

13. A data processing apparatus configured in a server, comprising: A processor, a memory, and a program thereon, the program causing the device to perform the functions of the system modules according to any one of claims 1-7.

Images