Low-code system for data analysis and data cleaning based on Internet of Vehicles data

Abstract

The invention provides a low-code system for data analysis and data cleaning based on Internet of Vehicles data. The low-code system comprises a multi-protocol adapter, a user-defined analysis rule writing interface, a dragging type rule configuration interface, a data processing module, a forwarding mechanism unit, a monitoring and log tracking unit and a message labeling and fingerprint generation unit. Wherein the multi-protocol adapter is used for supporting log data acquisition of various network protocols; the user-defined analysis rule writing interface allows a user to write specific analysis logic according to a data format and a service requirement; meanwhile, for compatibility of various data codes and formats, rule templates and examples are provided to guide a user to compile analysis rules, and the user is helped to quickly position and correct problems in the analysis rules by integrating corresponding error detection and debugging tools; and a user carries out graphical operation combination input, output and analysis on rule nodes through the drag-and-drop rule configuration interface.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle log data collection and processing, and in particular to a low-code system for data analysis and data cleaning based on Internet of Vehicles data. Background Art

[0002] At present, with the widespread application of vehicle information systems and vehicle-to-cloud communication systems, the types of log data collected and reported by the equipment are becoming increasingly diverse, covering key data such as operation status logs, safety event records, resource usage, etc. These data are uploaded to the cloud for centralized processing, providing strong support for services such as equipment health monitoring, abnormal event alarms, and vehicle and road condition analysis, helping enterprises achieve intelligent, data-driven operation and management.

[0003] During the log collection process, the device side usually uses network protocols such as TCP and MQTT to transmit data. However, since lightweight protocols such as MQTT lack functions such as persistent storage and advanced message management, and do not have the reliability of enterprise-level data management, most companies will convert and forward log data from protocols such as MQTT to familiar internal message middleware systems (such as Kafka, RabbitMQ, etc.) to achieve more stable message transmission and persistent storage. At the same time, third-party service providers need to be compatible with different protocol types to ensure seamless docking and stable transmission of log streams.

[0004] In terms of log parsing, in order to reduce traffic consumption between vehicle-mounted devices and cloud systems and improve data transmission efficiency, log data is usually encoded in binary format or Protocol Buffer (protobuf). The cloud system must define corresponding parsing rules based on the data format for decoding and data extraction. In addition, since different companies or systems may use customized middleware and protocol formats, such as JSON, XML, etc., the cloud system also needs to support multiple protocols and formats to flexibly adapt to different data input and output requirements. This diverse demand significantly increases the system's adaptability requirements for protocol conversion and format parsing.

[0005] The above complex requirements lead to the fact that in new projects, the system usually needs to be highly customized to meet the specific requirements of customers. This customized development not only involves the initial functional implementation, but also requires continuous operation and upgrades when customer needs are updated or business adjustments are made to maintain the efficiency, flexibility and reliability of the system. This high frequency of updates and maintenance brings significant technical challenges, including strict requirements for system scalability, deep optimization of multi-protocol compatibility, and stability assurance under large-scale data streams. There is an urgent need for a low-code system for data parsing and data cleaning based on Internet of Vehicles data. Summary of the invention

[0006] The present invention proposes a low-code system for data parsing and data cleaning based on Internet of Vehicles data, which solves the problems of complexity and lack of flexibility in the parsing and cleaning processes. By providing a graphical interface and multi-protocol support, the technical threshold for data processing is lowered and the processing efficiency is improved.

[0007] The technical solution of the present invention is implemented as follows: a low-code system for data parsing and data cleaning based on Internet of Vehicles data, including a multi-protocol adapter, a custom parsing rule writing interface, a drag-and-drop rule configuration interface, a data processing module, a forwarding mechanism unit, a monitoring and log tracking unit, and a message labeling and fingerprint generation unit; The multi-protocol adapter is used to support the log data collection of multiple network protocols; the custom parsing rule writing interface allows users to write specific parsing logic according to data formats and business requirements; and at the same time, for the compatibility of multiple data encodings and formats, rule templates and examples are provided to guide users to write parsing rules, and by integrating corresponding error detection and debugging tools, help users quickly locate and correct problems in parsing rules; users use the drag-and-drop rule configuration interface to graphically combine input, output and parsing rule nodes; the interface also has real-time preview and editing functions for rule nodes, and users can view and adjust configurations instantly, while performing version control of rule configurations, allowing multiple versions to be saved, rolled back and compared in the drag-and-drop rule configuration interface; a customized configuration tool is provided in the interface; the data processing module is used to perform field mapping, calculation and filtering operations; the forwarding mechanism unit forwards the processed data to the specified module according to the output rules configured by the user; the monitoring and log tracking system is used to track the collection, parsing and processing status of log data, and record system operation indicators; the message labeling and fingerprint generation unit generates a type label and a unique message fingerprint for each message, and performs data tracing through the unique message fingerprint.

[0008] Traditional data analysis and cleaning tools often rely on professional programming knowledge, and users need to write complex codes to implement data processing logic, which is a huge challenge for non-technical users and limits the popularity and application of data processing. However, this system introduces the low-code concept, allowing users to perform graphical operations through a drag-and-drop rule configuration interface, significantly reducing the technical threshold for data analysis and cleaning, allowing more users to participate in the data processing process.

[0009] The multi-protocol adapter function of this system allows users to support log data collection of multiple network protocols, which is not common in the existing technology. Traditional systems can only process data of specific protocols, which makes data integration between different devices and systems in the Internet of Vehicles environment complicated and inefficient. Through the design of multi-protocol adapters, this system can flexibly respond to various data sources, enhance the compatibility and applicability of the system, and enable it to better adapt to the diverse needs of the Internet of Vehicles environment.

[0010] In terms of writing custom parsing rules, traditional tools usually lack effective guidance, and users are prone to errors or omissions when writing parsing logic, resulting in inaccurate data processing. This system provides a custom parsing rule writing interface, allowing users to write specific parsing logic according to data formats and business requirements, and also provides rule templates and examples to guide users in writing parsing rules. This design not only improves user convenience, but also effectively reduces data processing problems caused by rule writing errors.

[0011] The system integrates error detection and debugging tools to help users quickly locate and correct problems in parsing rules. This function is relatively rare in existing technologies, and users usually need to debug the code manually, which consumes a lot of time and energy. However, this system greatly improves the efficiency and accuracy of data parsing through an intelligent error detection mechanism.

[0012] In terms of data processing modules, traditional systems often only provide basic field mapping and filtering functions, while this system enhances the ability of calculation and filtering operations, allowing users to flexibly process data in multiple dimensions to meet complex business needs. At the same time, the forwarding mechanism unit forwards the processed data to the specified module according to the output rules configured by the user, realizing efficient flow of data after processing. This efficient forwarding mechanism often requires complex configuration and programming in traditional tools, but this system simplifies this process through a graphical interface and improves the user experience.

[0013] The introduction of the monitoring and log tracking unit enables users to track the collection, analysis and processing status of log data in real time, and record system operation indicators. This function is often lacking in existing technologies, and it is difficult for users to understand the real-time operation status of the system. However, this system provides comprehensive visual feedback through monitoring and log tracking functions, helping users to discover and solve problems in a timely manner, and improving the reliability and stability of the system.

[0014] The message labeling and fingerprint generation unit generates a type label and a unique message fingerprint for each message, supporting data traceability. This function is particularly important in Internet of Vehicles application scenarios, and can effectively track the source of data and processing history, providing important support for subsequent data analysis and troubleshooting. In contrast, traditional systems often lack effective data traceability capabilities, making it difficult to track the true source and change history of data during data analysis and processing. This low-code system overcomes many of the shortcomings of traditional data parsing and cleaning tools through graphical operations, multi-protocol support, and intelligent function integration, demonstrating greater flexibility and ease of use, and providing a more efficient and reliable solution for Internet of Vehicles data processing.

[0015] As a preferred embodiment, the data processing module can perform field mapping through the user-defined correspondence between fields; when the user performs complex data conversion and calculation, the field data is mapped and calculated accordingly; at the same time, the data is filtered and the log data is screened according to user-defined conditions to ensure accurate and efficient data processing.

[0016] As a preferred implementation, the forwarding mechanism unit has multiple output targets and simultaneously forwards to the database, message queue, and file system; it also has dynamic routing rules and intelligently selects the best output path combination based on data attributes and business logic; it uses load balancing and failover strategies to balance the selection within the best output path combination to ensure the stability of data forwarding.

[0017] As a preferred embodiment, the monitoring and log tracking unit includes a monitoring panel, which displays key indicators and status information including operating frequency, concurrency, and delay; the monitoring and log tracking unit adopts a centralized storage and query method for saving and querying system operation logs; it has a built-in alarm and notification mechanism, which will issue an alarm according to the set path and simultaneously notify the user when an abnormality occurs.

[0018] As a preferred embodiment, the message labeling and fingerprint generation unit automatically generates type tags based on message content and attributes, wherein the message content includes a timestamp, a source address, and a target address; enhances data readability and query efficiency through type tags; and generates a unique message fingerprint based on the above content, and adopts the query and screening functions of tags and fingerprints for users to track and analyze messages, thereby enhancing log tracking and ensuring the integrity and traceability of messages.

[0019] After adopting the above technical solution, the beneficial effect of the present invention is that the system enables users to easily perform data analysis and cleaning by providing a drag-and-drop rule configuration interface, reducing the dependence on technical background. This feature enables non-technical users to participate in data processing, improving the work efficiency and overall productivity of the team. By simplifying the operation process, users can more quickly implement data analysis and cleaning tasks, thereby speeding up the data processing cycle and meeting rapidly changing business needs. The introduction of multi-protocol adapters enhances the compatibility of the system and can support data collection from different devices and systems. This design enables users to flexibly process various data sources in the Internet of Vehicles environment, avoiding the cumbersomeness and inefficiency of traditional tools in data integration. At the same time, the flexibility of the system enables users to quickly adapt to new data sources and formats, improving the adaptability of data processing. The custom parsing rule writing interface and error detection tools integrated in the system greatly improve the accuracy and efficiency of data parsing. When users write parsing rules, they can get timely feedback and help, thereby reducing data processing problems caused by rule errors. The implementation of this function improves the reliability of data processing, thereby improving the quality and accuracy of subsequent data analysis. The introduction of monitoring and log tracking functions provides users with real-time system operation status feedback. Users can keep abreast of data collection, parsing, and processing, identify and resolve potential problems, and ensure the stable operation of the system. This real-time monitoring capability improves the reliability of the system and reduces business losses caused by system failures. Message labeling and fingerprint generation units provide effective support for data traceability, allowing users to track the source and history of data. This function not only improves the transparency of data processing, but also provides an important basis for subsequent data analysis, auditing, and compliance, and enhances the credibility and security of the system. This low-code system provides strong technical support for Internet of Vehicles data processing by improving the efficiency, accuracy, and flexibility of data parsing and cleaning, significantly enhancing the company's capabilities in data management and decision analysis, and promoting the digital transformation and intelligent development of the company. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 It is a schematic diagram of the system flow of the present invention; DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Example: like Figure 1 As shown, the present invention provides a flexible vehicle log data collection and processing system to meet various data transmission, analysis and management requirements between vehicle-mounted devices and the cloud. The system has high adaptability and customization capabilities, can handle a variety of log types and protocols, and mainly includes the following core functions: Log data collection with multi-protocol support: The system supports log collection of multiple network protocols (such as TCP, MQTT, Kafka, and HTTP), and users can choose the appropriate protocol according to project requirements. For protocols that lack persistence and advanced message management functions (such as MQTT), the system is able to forward log streams to middleware that supports persistence (such as Kafka) to ensure the reliability of log data. This design allows the system to flexibly adapt to different network environments and business needs, while ensuring the stability and security of data transmission.

[0024] Custom parsing rules: Due to the high traffic cost between cars and the cloud, the system supports efficient compression encoding of log data (such as binary and protobuf formats), and allows users to write custom parsing rules in the cloud to decode data. In addition, the system is compatible with multiple data formats, including JSON, XML, etc., and can adapt to the customized needs of different enterprises and third-party middleware. This flexibility enables the system to handle various complex data formats, improve data transmission efficiency, and reduce operating costs.

[0025] Data processing and forwarding management: The system supports a variety of processing of collected log data, including field mapping, field calculation, and data filtering, to help users extract and analyze key information. At the same time, the system can forward the processed data to the specified storage or processing module according to the output rules configured by the user to support various data analysis, alarm and monitoring applications. This powerful data processing capability enables the system to meet various complex business needs and provides great convenience for users.

[0026] Real-time monitoring and log management: The system has a running log management function, which can monitor various indicators in the collection and analysis process in real time and record the log operation status. Through unified interface management, it can effectively track data flow and processing flow, improve the maintainability of the system and the transparency of operation. This real-time monitoring capability ensures that the system can discover and solve problems in a timely manner, ensuring the stable operation of the system.

[0027] Drag-and-drop rule configuration: The system provides an intuitive drag-and-drop rule configuration canvas. Users can freely combine input, output, parsing and other rule nodes through a graphical interface to achieve flexible configuration of data collection, processing and storage rules. Complex rules can be set without programming knowledge, which improves the convenience of operation and configuration efficiency. This intuitive operation method greatly reduces the user's learning cost, allowing non-technical personnel to easily manage and configure the system.

[0028] Message type tags and fingerprints: The system automatically assigns one or more type tags to each collected log message. These tags are based on the message's content, source, or destination. Type tags help quickly identify and classify messages, so that specific types of log information can be quickly located when processing large amounts of data. Each message also generates a unique message fingerprint that uniquely identifies the message content. This is used to quickly troubleshoot errors when parsing results are incorrect.

[0029] Tracking and troubleshooting: In system configuration, users can use type tags and message fingerprints to track the flow and processing of messages. When the system is abnormal or needs troubleshooting, users can quickly locate the problem message by querying specific tags or fingerprints, analyze its processing path and status, and thus speed up the diagnosis and resolution of the problem.

[0030] Through the above functions, the vehicle log data collection and processing system of the present invention can provide users with a comprehensive, efficient and reliable data management solution to meet the growing data processing needs of modern vehicle information systems.

[0031] The data processing module can perform field mapping by defining the corresponding relationship between fields through users; when users perform complex data conversion and calculation, they can map the field data to corresponding calculations; at the same time, by filtering the data, the log data can be screened according to the conditions defined by the user to ensure accurate and efficient data processing.

[0032] The data processing module can effectively support users to accurately map and calculate field data when performing complex data conversion and calculation through the function of field mapping. In a specific implementation scenario, assume that we are processing log data from multiple data sources, such as user behavior logs, system performance logs, and transaction logs. When using the data processing module, users first need to define the correspondence between fields, such as mapping the "user ID" field to the "customer ID", or converting the "timestamp" field to the "event occurrence time". The establishment of this mapping relationship lays the foundation for subsequent data processing.

[0033] After the user completes the field mapping definition, the data processing module will batch process the log data according to these mapping relationships. By performing the corresponding calculations and conversions, users can easily convert the raw data into a format that meets their analysis needs. For example, the user may want to summarize user activities within a specific time period. The data processing module can automatically calculate the number of activities for each user within that time period and output the results as a new data set. In addition, the data processing module also has a powerful data filtering function. Users can set filtering conditions, such as filtering out activity records of specific users or only retaining log data within a specific date range. In this way, the data processing module not only improves the accuracy of data processing, but also significantly improves work efficiency.

[0034] The forwarding mechanism unit has multiple output targets, and simultaneously forwards the database, message queue, and file system; it also has dynamic routing rules, and intelligently selects the best output path combination according to data attributes and business logic; it adopts load balancing and failover strategies to balance the selection within the best output path combination to ensure the stability of data forwarding. In terms of the forwarding mechanism unit, the system has designed multiple output targets, including databases, message queues, and file systems. Suppose that after a log processing is completed, the user wants to store the processing results in multiple locations at the same time for subsequent use. The forwarding mechanism unit will intelligently select the best output path based on the attributes and business logic of the data. For example, the processed data may need to be stored in a relational database for subsequent analysis, and it also needs to be pushed to the message queue for real-time processing by other services. The system automatically determines the forwarding path of the data through dynamic routing rules to ensure that each output target can receive the latest data in a timely manner.

[0035] In this process, load balancing and failover strategies ensure the stability of data forwarding. If a short network failure occurs during the process of forwarding data to the database, the system can quickly switch to the backup database connection to ensure that data is not lost. At the same time, the forwarding mechanism will dynamically adjust the output strategy according to the current load situation to ensure that each output target can run under the best conditions and avoid delays caused by excessive load.

[0036] The monitoring and log tracking unit includes a monitoring panel, which displays key indicators and status information including operating frequency, concurrency, and delay through the monitoring panel; the monitoring and log tracking unit adopts a centralized storage and query method to save and query the system operation log; it has a built-in alarm and notification mechanism, which alarms and notifies the user synchronously according to the set path when an abnormality occurs. The monitoring and log tracking unit plays a key role in the entire system. The unit has a built-in monitoring panel, which displays key indicators such as the operating frequency, concurrency, and delay of the system in real time. Users can intuitively understand the current status of the system through the panel. During the operation of the system, all operation logs will be centrally stored and queried for subsequent auditing and troubleshooting. When the system detects an abnormal situation, such as data processing delay exceeding the set threshold, the monitoring and log tracking unit will immediately trigger an alarm and synchronously notify the relevant users according to the preset path. Through this real-time monitoring and alarm mechanism, users can respond quickly and take measures to ensure the stable operation of the system.

[0037] The message labeling and fingerprint generation unit automatically generates a type label based on the message content and attributes, wherein the message content includes a timestamp, a source address, and a target address; the type label enhances the readability and query efficiency of the data; and generates a unique message fingerprint based on the above content, and uses the query and screening functions of the label and fingerprint for users to track and analyze messages, thereby enhancing log tracking and ensuring the integrity and traceability of the message. In addition, the message labeling and fingerprint generation unit significantly enhances the readability and query efficiency of the data by automatically generating a type label based on the message content and attributes. During the processing, the system generates a unique message fingerprint based on the timestamp, source address, and target address of each message. These fingerprints are not only used to ensure the integrity of the message, but also provide users with powerful tracking and analysis capabilities. Users can query and filter through labels and fingerprints to quickly locate the source and flow path of a specific message, thereby enhancing the accuracy of log tracking.

[0038] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A low-code system for data analysis and data cleaning based on Internet of Vehicles data, characterized in that: It includes multi-protocol adapter, custom parsing rule writing interface, drag-and-drop rule configuration interface, data processing module, forwarding mechanism unit, monitoring and log tracking unit, and message labeling and fingerprint generation unit; The multi-protocol adapter is used to support the collection of log data from multiple network protocols; the custom parsing rule writing interface allows users to write specific parsing logic based on data format and business requirements; At the same time, for the compatibility of multiple data encodings and formats, rule templates and examples are provided to guide users in writing parsing rules. By integrating corresponding error detection and debugging tools, users can quickly locate and correct problems in parsing rules. Users can graphically combine input, output and parsing rule nodes through the drag-and-drop rule configuration interface. The interface also has real-time preview and editing functions for rule nodes, allowing users to view and adjust configurations instantly, and version control of rule configurations is performed at the same time. Multiple versions can be saved, rolled back and compared in the drag-and-drop rule configuration interface. A customized configuration tool is provided in the interface; the data processing module is used to perform field mapping, calculation and filtering operations; the forwarding mechanism unit forwards the processed data to the specified module according to the output rules configured by the user; the monitoring and log tracking system is used to track the collection, parsing and processing status of log data, and record system operation indicators; the message labeling and fingerprint generation unit generates a type label and a unique message fingerprint for each message, and performs data tracing through the unique message fingerprint.

2. A low-code system for data analysis and data cleaning based on Internet of Vehicles data as claimed in claim 1, characterized in that: The data processing module can perform field mapping, and the corresponding relationship between the fields is defined by the user; When users perform complex data conversion and calculation, they map the field data to the corresponding calculation; at the same time, by filtering the data, the log data is screened according to the user-defined conditions to ensure accurate and efficient data processing.

3. A low-code system for data analysis and data cleaning based on Internet of Vehicles data as claimed in claim 1, characterized in that: The forwarding mechanism unit has multiple output targets and synchronously forwards to databases, message queues, and file systems; it also has dynamic routing rules and intelligently selects the best output path combination based on data attributes and business logic; it adopts load balancing and failover strategies and performs balanced selection within the best output path combination to ensure the stability of data forwarding.

4. A low-code system for data analysis and data cleaning based on Internet of Vehicles data as claimed in claim 1, characterized in that: The monitoring and log tracking unit includes a monitoring panel, which displays key indicators and status information including operating frequency, concurrency, and delay; the monitoring and log tracking unit adopts a centralized storage and query method to save and query system operation logs; it has a built-in alarm and notification mechanism, which will issue an alarm according to the set path and simultaneously notify the user when an abnormality occurs.

5. A low-code system for data analysis and data cleaning based on Internet of Vehicles data as claimed in claim 1, characterized in that: The message labeling and fingerprint generation unit automatically generates type tags based on message content and attributes, wherein the message content includes a timestamp, a source address, and a target address; type tags are used to enhance data readability and query efficiency; and a unique message fingerprint is generated based on the above content, and query and screening functions of tags and fingerprints are used for users to track and analyze messages, thereby enhancing log tracking and ensuring the integrity and traceability of messages.

Images