Internet of vehicles data processing method and system based on TTDengine

Abstract

The invention discloses an Internet of Vehicles data processing method and system based on TD engine, and the method comprises the following steps: S1, building a TD engine time sequence database, and carrying out the initialization; s2, modifying data related services of the Internet of Vehicles, and adding TD engine related configurations; s3, carrying out a multi-dimensional test on the TD engine time sequence database; and S4, monitoring and storing related performance indexes of the Internet of Vehicles data, and after the performance indexes are stabilized for a preset time, independently supporting data storage and query through a TD engine time sequence database. According to the method, the data processing efficiency, the data writing capability and the query performance are improved by using the TDengine, the dependence on an additional message queue system can be reduced, and the system complexity and the maintenance cost are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle networking data processing, and in particular to a vehicle networking data processing method and system based on TDengine. Background Art

[0002] The existing IoV technology framework often faces the challenges of massive data processing and real-time requirements. Traditional database systems often have problems such as low efficiency and long response time when processing continuously generated vehicle time series data (such as location information, speed, sensor readings, etc.), making it difficult to meet the dynamic data load and query requirements of the IoV platform.

[0003] The commonly used data storage solutions are as follows: using relational databases, such as MySQL; using non-relational KV databases, such as MongoDB, HBase. Problems with the existing technology: Relational databases have cluster throughput bottlenecks: The volume of Internet of Vehicles business data is huge, and the database throughput requirements are very high. Traditional MySQL cluster architectures often fail to meet the business needs of car companies. This means that when processing large amounts of Internet of Vehicles data, MySQL may not be able to provide sufficient performance and scalability.

[0004] Problems with multi-column index matching rules: In the Internet of Vehicles scenario, when the amount of data reaches a certain level, although MySQL can meet the business write needs, its multi-column index needs to meet the prefix matching to be effective. When the query conditions do not meet the multi-column index matching rules, performance degradation may occur. This shows that MySQL may have efficiency issues when processing complex query conditions, which is an important limitation for Internet of Vehicles applications that require efficient query of historical data and real-time data.

[0005] Inadaptability of traditional big data models and query languages: The data models of MongoDB and HBase are not suitable for processing time series data, which is particularly important in the field of Internet of Vehicles. Internet of Vehicles data usually contains a large amount of time series data, such as vehicle status, driving speed, location information, etc. These data require efficient and flexible storage and query methods to support real-time analysis and decision-making.

[0006] Traditional non-relational databases have a slow writing speed: When processing large amounts of time series data, the writing speed of MongoDB and HBase will become a bottleneck. In the Internet of Vehicles scenario, smart vehicles regularly collect vehicle status information through various sensors, generating a large amount of real-time data. If the database writing speed cannot keep up with the speed of data generation, it will seriously affect the efficiency and accuracy of data processing.

[0007] Traditional non-relational databases have problems with high operation and maintenance support costs and high storage costs: With the accelerated expansion of business, the problem of high support costs faced by MongoDB and HBase when processing Internet of Vehicles data has gradually emerged. This includes hardware costs, operation and maintenance costs, and possible technology upgrade costs, etc. For Internet of Vehicles applications that pursue cost-effectiveness, this is an issue that cannot be ignored. Taking Leapmotor as an example, storing time series data in MongoDB will result in excessively high storage costs, because MongoDB stores all data in memory and can only store data within a period of time. This high storage cost not only increases the difficulty of operation and maintenance, but also limits the time range of data storage, which is not conducive to long-term data analysis and mining.

[0008] Traditional non-relational databases have insufficient data lifecycle management: MongoDB and HBase rarely set up data lifecycle management, which is a disadvantage for Internet of Vehicles data that needs to be stored and analyzed for a long time. Internet of Vehicles data often has a certain historical value. Reasonable data lifecycle management can help companies save storage space and reduce operation and maintenance costs while ensuring data security and accessibility. In addition, traditional relational and non-relational databases also have problems such as query performance drops sharply when the query time span is large; do not support statistical analysis queries; do not support streaming computing scenarios, etc. Summary of the invention

[0009] In response to the above technical problems, the present invention provides a method and system for processing Internet of Vehicles data based on TDengine.

[0010] The present invention is implemented by adopting the following technical solution: a vehicle network data processing method based on TDengine, comprising the following steps: S1: Build the TDengine time series database and initialize it; S2: Modify the Internet of Vehicles data related services and add TDengine related configurations; S3: Perform multi-dimensional testing on the TDengine time series database; S4: Monitor and store performance indicators related to Internet of Vehicles data. After the performance indicators stabilize for a preset time, the TDengine time series database independently supports data storage and query.

[0011] Furthermore, step S1 includes the following sub-steps: S11: Build the TDengine time series database and apply for Internet of Vehicles data related resources; S12: Design a super table structure, create and initialize a data table to store Internet of Vehicles data.

[0012] Furthermore, step S2 includes the following sub-steps: S21: Modify the Internet of Vehicles data storage service configuration and add TDEngine storage configuration; S22: Modify the Internet of Vehicles data query service and add TDEngine query logic and configuration.

[0013] Furthermore, the multi-dimensional test includes one or more of a data cluster expansion and contraction test, a data node downtime scenario test, a stability test, and a performance test.

[0014] Furthermore, the expansion and reduction test of the data cluster includes: When testing the expansion and reduction of the data cluster, we simulate different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales. We simulate expansion and reduction by adding or reducing data nodes, and monitor the response time and resource consumption of the TDengine time series database.

[0015] Furthermore, the data node downtime scenario test includes: When testing data node downtime scenarios, we simulate different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales. We simulate capacity expansion and contraction by adding or reducing data nodes, while monitoring the response time and resource consumption of the TDengine time series database.

[0016] Further, the stability test includes: When conducting stability tests, we simulate different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales. We simulate capacity expansion and contraction by adding or reducing data nodes, and monitor the response time and resource consumption of the TDengine time series database. The stability test also includes: designing test cases to cover various factors that may affect system stability, including memory usage and connection pool control, and recording test results for analysis and improvement.

[0017] Furthermore, the performance test includes: using TSBS and taosBenchmark tools to perform detailed performance comparison tests.

[0018] Furthermore, the performance indicators include one or more of IoT export data accumulation, database data storage growth, database server CPU, memory, disk space usage, and data query performance.

[0019] A TDengine-based Internet of Vehicles data processing system is used to implement the above-mentioned TDengine-based Internet of Vehicles data processing method, including a database building module, an appending module, a testing module and a monitoring module, wherein: Database building module, used to build the TDengine time series database and initialize it; Add modules to modify Internet of Vehicles data-related services and add TDengine-related configurations; The test module is used to perform multi-dimensional tests on the TDengine time series database; The monitoring module is used to monitor and store performance indicators related to Internet of Vehicles data. After the performance indicators stabilize for a preset time, the TDengine time series database independently supports data storage and query.

[0020] The beneficial effects of the present invention are: using TDengine to improve data processing efficiency, data writing capability and query performance; reducing dependence on additional message queue systems, reducing system complexity and maintenance costs; high throughput and low latency characteristics are suitable for high-speed data stream processing; powerful analytical query language facilitates complex time series data analysis; significant advantages lie in the multi-level protection of network security, data security and communication security. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0022] Figure 1 This is a schematic diagram of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0024] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0025] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] See also Figure 1 , a vehicle networking data processing method based on TDengine, comprising the following steps: S1: Build the TDengine time series database and initialize it; S2: Modify the Internet of Vehicles data related services and add TDengine related configurations; S3: Perform multi-dimensional testing on the TDengine time series database; S4: Monitor and store performance indicators related to Internet of Vehicles data. After the performance indicators stabilize for a preset time, the TDengine time series database independently supports data storage and query.

[0027] In this embodiment, step S1 includes the following sub-steps: S11: Build the TDengine time series database and apply for Internet of Vehicles data related resources; S12: Design a super table structure, create and initialize a data table to store Internet of Vehicles data.

[0028] To build a TDengine time series database, you need to consider the following aspects: scenario analysis, technology selection, and database selection. Scenario analysis: The current business scenarios of the Internet of Vehicles and the temporal characteristics of Internet of Vehicles data are the most important prerequisites for technology selection. Data is not updated or deleted: Internet of Vehicles data is different from traditional business data. Most of the data reported by vehicles are reported to the Internet of Vehicles platform at a certain frequency. These data have strong consistency with time. After the data is reported, they will not be repeatedly updated or deleted. Therefore, technology selection only needs to focus on the scenarios of storage and query. Stability and non-transactional: Internet of Vehicles data is reported one by one at a fixed frequency, and the data flow is stable, without the need for traditional database transaction processing. Obvious query and analysis characteristics: Most of the businesses based on Internet of Vehicles data are query and analysis based on time periods and spatial regions, such as real-time monitoring, historical trajectory playback, regional vehicle search, etc. Streaming computing scenarios: In addition to the above-mentioned businesses with strong temporal characteristics, statistics and real-time computing operations in various dimensions are also required for Internet of Vehicles data. For example: driving behavior analysis based on location information, stop and go statistics and other businesses. Ultra-large-scale data and increments: Internet of Vehicles data is reported frequently, and the base of existing vehicles is large. Based on the average reporting frequency of one location information data item every 10 seconds, the IoV platform collects 8,640 data items per vehicle per day. If an IoV monitoring platform is connected to 10,000 vehicles, the platform collects more than 80 million data items per day, and the data will increase with the growth of business.

[0029] Technology selection: According to the business scenarios and the time series characteristics of data, time series databases have obvious advantages in processing time series data of the Internet of Vehicles, including but not limited to high write and fast query capabilities, efficient column storage, powerful massive data processing capabilities, cost-effectiveness, real-time data analysis capabilities, and good scalability and flexibility. These characteristics make time series databases the best choice for storing time series data of the Internet of Vehicles. High write and fast query requirements: The amount of GPS data generated by the Internet of Vehicles is huge, reaching trillions, which requires the data storage system to support high write speed and fast data query. Time series databases have higher efficiency and performance in processing time series data and can meet these basic requirements. Column storage advantages: Time series databases use column storage, which has significant advantages in compression rate, space saving and query performance. Since time series data usually contains a large amount of identical or similar data, especially timestamps fields, column storage can effectively improve data compression rate and query efficiency. Massive data processing capabilities: The amount of data in the Internet of Vehicles scenario is huge, requiring a system that can handle PB-level data storage and support high writes of tens of millions per second. Time series databases just meet this demand. They have a simple architecture and can complete the required functions by using tables to store time series tables. They also support metadata retrieval and time series data retrieval. Cost-effectiveness: With the accelerated expansion of business, traditional storage solutions such as MongoDB or Apache HBase may face problems such as slow write speeds and high support costs. However, time series databases can reduce storage costs and improve economic benefits while ensuring performance through optimized data models and storage strategies. Real-time data analysis capabilities: Time series databases not only support high-speed data processing and large-scale data queries, but also have the ability to analyze data in real time. This is especially important for application scenarios such as the Internet of Vehicles, because real-time data analysis can help companies obtain valuable information in a timely manner and make more informed decisions. Scalability and flexibility: Time series databases have good scalability and flexible data models, which can adapt to the changes and development needs of the business characteristics of the Internet of Vehicles. This makes time series databases an ideal choice for Internet of Vehicles data storage.

[0030] Database selection: There are many time series databases on the market, and each database has its own unique advantages. Considering the comprehensive functional characteristics, learning cost, storage space and cost, reliability and other aspects, the present invention makes the following choices: OpenTSDB: Its essence is to add a layer of cluster on the basis of the existing HBase. Its structural system is larger than the Hbase architecture and has higher storage costs. InfluenceDB: The cluster version is a paid resource, and the community version does not have a cluster solution. From the production environment point of view, high availability of data storage and query is necessary, so it is best to use the cluster version, so the operation and maintenance cost of this database is relatively high. CTSDB: A product of Tencent Cloud, this database provides functions such as efficient reading and writing, low-cost storage, and powerful aggregation analysis capabilities, but it is paid according to the amount of stored data. Therefore, considering the incremental data in the Internet of Vehicles scenario, the operation and maintenance cost is also very high. TDEngine: Supports JDBC3.0 specification, low code intrusion, and is compatible with MySQL syntax queries. It claims to have the advantages of zero learning cost and zero operation and maintenance cost, which is very attractive for rapid deployment and reducing long-term maintenance costs; it provides a multi-level storage solution that can significantly reduce storage costs; it supports arbitrary dimensional queries, statistical aggregation analysis, and simple streaming calculations; it supports caching the latest data, can replace Redis, and simplify the architecture; the data compression ratio is super high, and the disk occupancy is very small; the simple expansion and reduction mechanism does not need to consider data migration, which greatly reduces operation and maintenance costs.

[0031] Based on the above analysis, TDengine has the following characteristics, making it the data storage solution for the Internet of Vehicles platform recommended by the present invention: Functional characteristics: efficient reading and writing; powerful aggregation analysis capabilities; visualization of instance monitoring and data query results; support for near-real-time data search and analysis; distributed and scalable; high reliability. Storage cost: lower storage cost than OpenTSDB; multi-level storage reduces storage cost waste. Learning cost: zero operation and maintenance cost, zero learning cost. Performance comparison: write performance is better than the other databases mentioned above. Application scenarios: applicable to massive time series data scenarios such as the Internet of Things, big data, and Internet monitoring. Technical advantages: based on distributed high-reliability architecture design; supports horizontal expansion to meet large-scale data processing needs. Cost-effectiveness: the cost of hardware or cloud services is reduced to 1 / 5, which greatly reduces the waste of storage costs.

[0032] In this embodiment, step S2 includes the following sub-steps: S21: Modify the Internet of Vehicles data storage service configuration and add TDEngine storage configuration; S22: Modify the Internet of Vehicles data query service and add TDEngine query logic and configuration.

[0033] In this embodiment, the multi-dimensional test includes one or more of the following: data cluster expansion and contraction test, data node downtime scenario test, stability test, and performance test. The expansion and reduction test of the data cluster includes: when conducting the expansion and reduction test of the data cluster, by simulating different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales, by adding or reducing data nodes to simulate expansion and reduction, and monitoring the response time and resource consumption of the TDengine time series database.

[0034] The data node downtime scenario test includes: when testing the data node downtime scenario, by simulating different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales, by adding or reducing data nodes to simulate capacity expansion and contraction, and at the same time monitoring the response time and resource consumption of the TDengine time series database.

[0035] The stability test includes: when conducting stability testing, by simulating different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales, by adding or reducing data nodes to simulate capacity expansion and contraction, while monitoring the response time and resource consumption of the TDengine time series database; the stability test also includes: designing test cases to cover various factors that may affect system stability, the factors affecting system stability include memory usage and connection pool control, and recording test results for analysis and improvement.

[0036] The performance test includes: focusing on the writing and query performance of TDengine in different scenarios, and using TSBS and taosBenchmark tools to conduct detailed performance comparison tests.

[0037] In this embodiment, the performance indicators include one or more of IoT export data accumulation, database data storage growth, database server CPU, memory, disk space usage, and data query performance.

[0038] In this embodiment, step S4 is specifically as follows: monitoring indicators such as the accumulation of IoT export data, the growth of database data storage, the CPU, memory, and disk space usage of the database server, and data query performance. After the performance indicators have been running stably for a period of time, the original storage solution (HBase) is disconnected, and TDEngine is used to completely independently support data storage and query, and back up the original disk data.

[0039] A TDengine-based Internet of Vehicles data processing system is used to implement the above-mentioned TDengine-based Internet of Vehicles data processing method, including a database building module, an appending module, a testing module and a monitoring module, wherein the database building module is used to build a TDengine time series database and initialize it; the appending module is used to modify Internet of Vehicles data-related services and append TDengine-related configurations; the testing module is used to perform multi-dimensional testing on the TDengine time series database; the monitoring module is used to monitor and store performance indicators related to Internet of Vehicles data, and after the performance indicators are stable for a preset time, the TDengine time series database independently supports data storage and query.

[0040] Based on the above embodiments, the present invention has at least the following technical effects: Use TDengine to improve data processing efficiency, data writing capabilities and query performance; reduce dependence on additional message queue systems, reduce system complexity and maintenance costs; high throughput and low latency characteristics are suitable for high-speed data stream processing; powerful analytical query language facilitates complex time series data analysis; significant advantages lie in the multi-level protection of network security, data security and communication security.

[0041] For the aforementioned embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily required by the present application.

[0042] The above embodiments describe the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the changes and modifications made by those skilled in the art shall be within the scope of protection of the appended claims of the present invention without departing from the spirit and scope of the present invention.

Claims

1. A vehicle networking data processing method based on TDengine, characterized in that: The steps include: S1: Build the TDengine time series database and initialize it; S2: Modify the Internet of Vehicles data related services and add TDengine related configurations; S3: Perform multi-dimensional testing on the TDengine time series database; S4: Monitor and store performance indicators related to Internet of Vehicles data. After the performance indicators stabilize for a preset time, the TDengine time series database independently supports data storage and query.

2. A TDengine-based vehicle networking data processing method as claimed in claim 1, characterized in that: Step S1 includes the following sub-steps: S11: Build the TDengine time series database and apply for Internet of Vehicles data related resources; S12: Design a super table structure, create and initialize a data table to store Internet of Vehicles data.

3. A TDengine-based vehicle networking data processing method as claimed in claim 2, characterized in that: Step S2 includes the following sub-steps: S21: Modify the Internet of Vehicles data storage service configuration and add TDEngine storage configuration; S22: Modify the Internet of Vehicles data query service and add TDEngine query logic and configuration.

4. A method for processing Internet of Vehicles data based on TDengine as claimed in claim 3, characterized in that: The multi-dimensional test includes one or more of a data cluster expansion and reduction test, a data node downtime scenario test, a stability test, and a performance test.

5. The TDengine-based vehicle networking data processing method according to claim 4, characterized in that: The expansion and reduction test of the data cluster includes: When testing the expansion and reduction of the data cluster, we simulate different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales. We simulate expansion and reduction by adding or reducing data nodes, and monitor the response time and resource consumption of the TDengine time series database.

6. A TDengine-based vehicle networking data processing method as claimed in claim 4, characterized in that: The data node downtime scenario test includes: When testing data node downtime scenarios, we simulate different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales. We simulate capacity expansion and contraction by adding or reducing data nodes, and monitor the response time and resource consumption of the TDengine time series database.

7. The TDengine-based vehicle networking data processing method according to claim 4, characterized in that: The stability test includes: When conducting stability tests, we simulate different data volumes and query requests to evaluate the performance and stability of the TDengine time series database at different scales. We simulate capacity expansion and contraction by adding or reducing data nodes, and monitor the response time and resource consumption of the TDengine time series database. The stability test also includes: designing test cases to cover various factors that may affect system stability, including memory usage and connection pool control, and recording test results for analysis and improvement.

8. The TDengine-based vehicle networking data processing method according to claim 4, characterized in that: The performance test includes: using TSBS and taosBenchmark tools to perform detailed performance comparison tests.

9. The TDengine-based vehicle networking data processing method according to claim 1, characterized in that: The performance indicators include one or more of IoT export data accumulation, database data storage growth, database server CPU, memory, disk space usage, and data query performance.

10. A TDengine-based Internet of Vehicles data processing system, used to implement a TDengine-based Internet of Vehicles data processing method according to any one of claims 1 to 9, characterized in that: It includes database building module, appending module, testing module and monitoring module, among which, Database building module, used to build the TDengine time series database and initialize it; Add modules to modify Internet of Vehicles data-related services and add TDengine-related configurations; The test module is used to perform multi-dimensional tests on the TDengine time series database; The monitoring module is used to monitor and store performance indicators related to Internet of Vehicles data. After the performance indicators stabilize for a preset time, the TDengine time series database independently supports data storage and query.

Images