Streaming data processing method and device and related equipment

By analyzing and pushing the vehicle-side data to the kafka streaming data processing platform, wide table data for event time partitions are generated, massive data processing and query problems are solved, and efficient data acquisition and storage required for intelligent automobile development is realized.

CN119094549APending Publication Date: 2024-12-06Z-ONE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411103693.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing technology is difficult to support the rapid access and efficient processing and analysis of massive vehicle-side data, which leads to inconvenience in data queries in the data warehouse and is difficult to meet the data needs in the intelligent vehicle development process.

Method used

By parsing the ciphertext data generated by the car end into plaintext, it is pushed to the intermediate kafka stream data processing platform to generate wide table data partitioned according to event time, it supports real-time data acquisition and storage in OSS object services, and optimizes the generation of data warehouses and query process.

Benefits of technology

It realizes timely entry and efficient processing of massive vehicle-side data, simplifies the data acquisition process of downstream business parties, and reduces the storage cost and query complexity of data warehouses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094549A_ABST
    Figure CN119094549A_ABST
Patent Text Reader

Abstract

The invention provides a streaming data processing method and device and related equipment, and the method comprises the steps: analyzing ciphertext to-be-stored data generated by a vehicle end into plaintext to-be-stored data, and then pushing the plaintext to-be-stored data to an intermediate kafka streaming data processing platform; the intermediate kafka stream data processing platform is used for processing plaintext data to be stored, first wide table data and / or second wide table data are / is generated, and the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data for setting a rolling window according to event time, and the second wide table data is wide table data for completely representing vehicle end event data; and supporting a downstream business party to call data through the first wide table data and the second wide table data. According to the method, the problems of complexity of a signal format of the warehousing data, data transmission delay and the like do not need to be considered, massive data can be efficiently processed and analyzed, timely warehousing of the vehicle-end data is realized, and a downstream business party can conveniently retrieve and use the vehicle-end data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of vehicle networking technology, and in particular to a streaming data processing method, device and related equipment. Background Art

[0002] With the development of Internet of Vehicles technology, more and more data are transmitted in Internet of Vehicles applications, and the traffic is also increasing. In particular, in response to current application scenarios such as intelligent Internet of Vehicles data, various event data generated during vehicle driving are quickly calculated and processed, and historical event data are flushed back. Data warehouses or data warehouses for vehicle data have been formed, which provide a prerequisite for efficient and accurate analysis of Internet of Vehicles data, data mining and vehicle simulation, and other data applications for downstream vehicle companies or industries. With the rapid development of the smart car industry, it has become an increasingly urgent problem for every automobile company to obtain complete data on the vehicle side conveniently and efficiently, and provide various data required for the production and function development of smart cars. However, due to the complexity of vehicle signal formats, the huge amount of data, and data transmission delays, the data warehouse generation methods currently used in the industry are difficult to support the rapid access to massive vehicle-side data, and timely and efficient processing and analysis of massive data generated by the vehicle side, resulting in inconvenience in querying the data in the generated data warehouse, and it is difficult to meet the data needs of downstream business parties in the vehicle manufacturing industry in the process of developing smart cars. Summary of the invention

[0003] In view of this, embodiments of the present application provide a streaming data processing method, device, and storage medium to at least partially solve the above-mentioned problems.

[0004] In a first aspect, an embodiment of the present application provides a streaming data processing method, including:

[0005] After parsing the encrypted data to be stored generated by the vehicle into plaintext data to be stored, it is pushed to the intermediate Kafka stream data processing platform;

[0006] The plaintext data to be stored is processed by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that completely represents vehicle-side event data;

[0007] The first wide table data is used to support data calls of real-time data acquisition work scenarios of downstream business parties; and / or the second wide table data is stored in OSS object service storage to support data calls by downstream business parties.

[0008] Optionally, in an embodiment of the present application, the processing of the plaintext data to be stored by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data includes:

[0009] When generating the first wide table data, the intermediate Kafka stream data processing platform is used to process the plaintext data to be stored, and a data detail table partitioned by processing time is generated;

[0010] Based on the data detail table, an hourly scheduling strategy is executed for non-delayed data, and / or a batch scheduling strategy is executed for delayed data to generate the second wide table data.

[0011] Optionally, in an embodiment of the present application, the method further includes: determining, from the generated second wide table data, called cold data whose utilization rate is lower than a preset rate; and transferring the cold data to an OSS object storage service.

[0012] Optionally, in one embodiment of the present application, after parsing the ciphertext data to be stored into plaintext data to be stored, the data is pushed to the intermediate Kafka stream data processing platform, including: based on the necessary signal information determined by the data requirements of the downstream business, the plaintext data to be stored is converted into preset format data; the format data obtained after the conversion is pushed to the intermediate Kafka stream data processing platform. In this way, while ensuring that the data stored in the data warehouse meets the business development needs of the downstream business party, it also saves the required data warehouse space, and also facilitates the downstream business party to directly obtain data for use without further processing, thereby reducing the work links of the downstream business party's data use process and improving the efficiency of data acquisition.

[0013] Optionally, in an embodiment of the present application, the method further includes: storing the encrypted data to be stored in the database to be stored in an OSS object storage service for downstream business demanders to query and obtain required data.

[0014] Optionally, in an embodiment of the present application, the method further includes: generating a signal export dashboard corresponding to the encrypted data to be stored, so as to support downstream business demanders to query the data stored in the OSS object storage service.

[0015] Optionally, in an embodiment of the present application, the processing of the plaintext data to be stored by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data includes:

[0016] Using the intermediate Kafka stream data processing platform to perform ETL aggregation operations on the plaintext data to be stored, and obtaining necessary data to be stored; generating second wide table data corresponding to the necessary data to be stored;

[0017] Correspondingly, the storing of the second wide table data in the OSS object service storage for supporting the downstream business party to call data includes: performing statistics on the demand data of the downstream business party, and determining to obtain the indicator information in real time or obtain the statistical information offline;

[0018] Based on the real-time acquisition of indicator information or offline acquisition of statistical information, the call is made in combination with the second wide table data stored in the OSS object service storage.

[0019] Optionally, in an embodiment of the present application, the method further includes: setting a preset life cycle for the first wide table data, and / or setting a life cycle for the data detail table partitioned by processing time.

[0020] In a second aspect, based on the streaming data processing method described in the first aspect of the present application, the present application further provides a streaming data processing device, including:

[0021] The push module is used to parse the encrypted data to be stored generated by the vehicle into plaintext data to be stored, and then push it to the intermediate Kafka stream data processing platform;

[0022] A generation module, used to process the plaintext data to be stored by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that completely represents vehicle-side event data;

[0023] The storage module is used to support the data call of the real-time data acquisition work scenario of the downstream business party through the first wide table data; and / or store the second wide table data in the OSS object service storage to support the downstream business party to make data calls.

[0024] In a third aspect, an embodiment of the present application also provides a computer storage medium having computer executable instructions stored thereon, and when the computer executable instructions are executed, any one of the streaming data processing methods described in the first aspect of the present application is performed.

[0025] In a fourth aspect, an embodiment of the present application further provides an electronic device, including:

[0026] one or more processors;

[0027] a memory for storing one or more programs,

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the streaming data processing methods described in the first aspect of the present application.

[0029] The present application provides a streaming data processing method, apparatus and related equipment, which parses the ciphertext data to be stored generated by the vehicle side into plaintext data to be stored, and pushes the data to an intermediate Kafka streaming data processing platform; uses the intermediate Kafka streaming data processing platform to process the plaintext data to be stored, and generates first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that fully represents the vehicle-side event data; through the first wide table data, supports data calls of real-time data acquisition work scenarios of downstream business parties; and / or, stores the second wide table data in OSS object service storage to support data calls by downstream business parties. The Willow data processing method provided in the embodiment of the present application uses an intermediate Kafka stream data processing platform to process the data generated by the vehicle side, and then generates wide table data partitioned by event time for default data and non-default data, and generates a data warehouse to support the vehicle development needs of downstream business parties. The data warehousing process does not require the data warehouse generator to consider issues such as the complexity of the signal format of the warehoused data and data transmission delays. It can efficiently process and analyze massive amounts of data, realize timely warehousing of vehicle-side data, and facilitate downstream business parties to retrieve and use it. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0031] Figure 1 A schematic diagram of a workflow of a streaming data processing method provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of the structure of a streaming data processing device provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0035] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0036] Embodiment 1

[0037] The present application embodiment provides a streaming data processing method, such as Figure 1 As shown, Figure 1 A flow chart of a streaming data processing method provided in an embodiment of the present application, the streaming data processing method comprising:

[0038] Step S101, after parsing the ciphertext data to be stored generated by the vehicle end into plaintext data to be stored, push it to the intermediate Kafka stream data processing platform. In the application process of the vehicle, the vehicle end generates mostly ciphertext data with high complexity, different formats and encrypted. In order to improve the convenience of these data being called, they can be decrypted first to obtain the corresponding plaintext data to be stored, and then push these plaintext data to the intermediate Kafka stream data platform for analysis and sorting, so as to avoid the need to consume the processing resources of the calling end to parse and analyze the data when the data is directly stored in the warehouse and called, and improve the convenience of data being called. In an embodiment of the present application, the intermediate Kafka stream data processing platform is a distributed open source stream data processing platform based on a message queue (MQ, Message Queue) in a publish / subscribe mode, which is mainly used in the field of real-time processing of big data. Its biggest feature is that it can process a large amount of data in real time to meet various demand scenarios, such as a batch processing system based on Hadoop and a low-latency real-time system. In an embodiment of the present application, the vehicle data to be stored in the warehouse is pushed to the platform for processing, saving processing resources when the data is called.

[0039] S102, using the intermediate kafka stream data processing platform to process the plaintext data to be stored, generating first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that fully represents vehicle-side event data. In an embodiment of the present application, a wide table is a kind of data warehouse, which refers to a database table in which indicators, dimensions, and attributes related to business topics are associated together. The data content contained in it is relatively comprehensive, although relatively redundant, but the time-partitioned wide table allows the database engine to query only partitions within a specific time range without having to scan the entire table. This can significantly improve query performance, especially when it is necessary to retrieve recent data or data within a specific time period. That is, wide table data partitioned by events has the advantages of efficient and convenient query. In addition, it is also relatively easy to manage and maintain time-partitioned wide table data. For example, the process of deleting old data, backing up or archiving data, and performing other maintenance operations can be more efficient. When time-partitioned wide tables are used to support fast insert operations, they only need to locate the appropriate time partition through the database engine without searching the entire table, which can significantly improve the processing of real-time data. At the same time, event-partitioned wide tables also support more sophisticated data retention policies based on time. For example, operation and maintenance personnel can easily delete or migrate old data without affecting the data in other partitions of the table. Through time partitioning, the performance of the database is more predictable in time, which is convenient for users to plan data and optimize queries. For scenarios that require archiving or historical data analysis, time-partitioned wide tables can easily support these requirements. It also supports users to selectively operate partitions within a specific time range without involving the data in the entire table. In the data-driven world, data analysis plays a vital role. Data warehouses that store data in the form of wide tables can help application developers extract valuable information from massive data and make correct decisions. In the process of data analysis, a wide table is a table structure that stores data horizontally. It stores related data in a row, making data access more efficient. Compared with the traditional narrow table structure, wide tables can provide more information and dimensions, thus providing data analysts with a more comprehensive and in-depth analysis perspective. Wide tables, with their simplicity, flexibility and efficiency, provide important tools and techniques for data analysts. Secondly, data warehouses in the form of wide tables support more flexible data pivoting and aggregation. In the process of data analysis, it is often necessary to pivot and aggregate data. The data storage structure of wide tables can make these operations simpler and more efficient.Through perspective and aggregation operations, data demanders can slice and summarize data according to different dimensions, time windows or other key factors when performing data analysis, so as to gain in-depth understanding of the changing trends and influencing factors of the data, and provide strong support for business decisions. Among them, grouping data based on time is a typical operation in data processing, which is when processing infinite streams. Time-based grouping is called a window, and the most basic window type is called a rolling window (Tumbling Windows), which is a way to divide data evenly. There is no overlap or gap between windows, and they are "connected end to end". When data needs to be aggregated on the dimension of a time-based rolling window for data call, Flink's window operator provides a convenient and easy-to-use API that can divide the data stream into windows to process the data in the window.

[0040] The embodiment of the present application generates a data warehouse in the form of a wide table through an intermediate Kafka stream data processing platform, which can more efficiently and completely ensure that data of various formats generated on the vehicle side are archived based on event time, wherein the event time is the time when the data is generated, that is, the timestamp information carried by the data generated during the working process of a certain function during the use of the vehicle.

[0041] S103. Support the data call of the real-time data acquisition work scenario of the downstream business party through the first wide table data; and / or store the second wide table data in the OSS object service storage to support the downstream business party to make data calls. In the embodiment of the present application, the generated first wide table data is mainly used to meet the data call requests generated by the real-time business scenario with high timeliness requirements of the downstream business party, but relatively low data integrity or data quality requirements. The second wide table data is mainly used to meet the data call requests generated by the business scenario with low timeliness requirements of the downstream business party, but relatively high data integrity or quality requirements. The OSS object storage service (Object Storage Service, OSS) is a massive, secure, low-cost, highly reliable cloud storage service suitable for storing any type of files. It has large capacity, supports elastic expansion of data processing capabilities, provides multiple storage types, and stores the generated wide table data through the OSS object storage service to comprehensively optimize storage costs.

[0042] Optionally, in an embodiment of the present application, the use of the intermediate kafka stream data processing platform to process the plaintext data to be stored to generate the first wide table data and / or the second wide table data includes: when generating the first wide table data, the use of the intermediate kafka stream data processing platform to process the plaintext data to be stored to generate a data list partitioned by processing time; based on the data list, the strategy of hourly scheduling is executed for non-delayed data, and / or the strategy of batch scheduling is executed for delayed data to generate the second wide table data. In an embodiment of the present application, the processing time is the time for parsing the data into the warehouse. In actual application scenarios, it is difficult to process all the data to be entered into the warehouse in time and quickly enter the warehouse due to various reasons such as the complexity of the data to be entered into the warehouse, the huge amount of data, and the transmission delay. In order to better deal with this situation, the embodiment of the present application can generate a corresponding partition table according to the current processing time for the data that needs to be processed in time and entered into the warehouse. The partition table is a detailed table containing data with good comprehensiveness and is stored in hive. Hive is a data warehouse based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism that supports storage, query and analysis of large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files to a database table and provide SQL query functions. It can convert SQL statements into MapReduce tasks for execution, which is used to support downstream business parties to conduct comprehensive query and acquisition of data with high timeliness. Furthermore, the intermediate Kafka stream data processing platform implements an hourly scheduling strategy for non-delayed data in the detailed table partitioned by events, that is, the corresponding wide table data partitioned by event time is formed at the same time as the data is generated to fall into the data warehouse. For delayed data, batch scheduling can be adopted. For example, when the amount of data generated on the vehicle side is small, or the transmission channel consumption required for data storage is small, or when the hardware processing resource utilization rate of the intermediate Kafka stream data processing platform is low, these delayed data are processed to generate wide table data partitioned by event time without default data to fall into the data warehouse, which is used to support the data usage needs of downstream business parties in the business development process. In addition, this method can not only meet the real-time needs of downstream business parties to obtain the data, but also better ensure the comprehensiveness and integrity of the warehouse data.Optionally, in the embodiment of the present application, the processing of data mainly includes determining the source of data generation and attribute analysis, data filtering, eliminating data redundancy, data merging or other ETL processing processes, that is, such as data Extraction, Transformation, Loading, etc., extracting data EXTRACTION from different data sources, performing format conversion TRASFORMATION on the extracted data according to certain data processing rules, and finally storing the processed data in the wide table of the data warehouse. The embodiment of the present application is not repeated here.

[0043] Optionally, in an embodiment of the present application, the method further includes: determining the cold data whose utilization rate is lower than a preset rate from the generated second wide table data, and transferring the cold data to an OSS object storage service. In the embodiment of the present application, the second wide table data generally falls into Hive to form a data warehouse for supporting downstream business parties to access data, and the infrequently used cold data in the Hive data warehouse is transferred to the OSS object storage service, which reduces the amount of data that needs to be stored through Hive, so as to save storage resources and reduce the storage cost of the generated data warehouse.

[0044] Optionally, in one embodiment of the present application, after the ciphertext data to be stored is parsed into plaintext data to be stored, it is pushed to the intermediate Kafka stream data processing platform, including: based on the necessary signal information determined by the data requirements of the downstream business, the plaintext data to be stored is converted into preset format data; the format data obtained after the conversion is pushed to the intermediate Kafka stream data processing platform. This is to facilitate the direct use of the downstream business party, shortening the working link of the downstream business party to call the data in the data warehouse for business development.

[0045] Optionally, in one embodiment of the present application, the method further includes: storing the encrypted data to be stored in the database to be stored in the OSS object storage service for downstream business demand parties to query and obtain the required data. The OSS object storage service is a storage service database with a usage cost. The acquired input to be stored is stored in the OSS object storage service, which is used as a full supplement to the data warehouse generated in Hive as the generated first wide table data or second wide table data. That is, if the downstream business party cannot query the target data from the wide table data in hive, the target data is queried through the OSS object storage service, so that the downstream business party can obtain the comprehensiveness of the data.

[0046] Optionally, in an embodiment of the present application, the method further includes: generating a signal export dashboard corresponding to the encrypted data to be stored, for supporting downstream business demand parties to query the data stored in the OSS object storage service. The signal export dashboard is a visual management tool for exporting signal data generated by the vehicle end, which provides a window interface for interacting with users. In this way, the embodiment of the present application supports users or downstream business parties to query data stored in the OSS object storage service, and the process is convenient, simple and easy to implement.

[0047] Optionally, in one embodiment of the present application, the intermediate kafka stream data processing platform is used to process the plaintext data to be stored to generate the first wide table data and / or the second wide table data, including: using the intermediate kafka stream data processing platform to perform ETL aggregation operations on the plaintext data to be stored to obtain the necessary data to be stored; generating the second wide table data corresponding to the necessary data to be stored. Correspondingly, the second wide table data is stored in the OSS object service storage to support the downstream business party to make data calls, including: statistics on the demand data of the downstream business party, and determine the real-time acquisition of indicator information or offline acquisition of statistical information; based on the real-time acquisition of indicator information or offline acquisition of statistical information, combined with the second wide table data stored in the OSS object service storage for calling. In this embodiment of the present application, by obtaining the actual indicator information and offline acquisition of statistical information of the downstream business party to obtain the data, resources are reasonably allocated to process the band storage data to generate the second wide table data differentiated by event time, and the second wide table data is stored in the OSS storage service to save the demand data of the downstream business party in a cost-saving manner.

[0048] Optionally, in an embodiment of the present application, the method further includes: setting a preset life cycle for the first wide table data, and / or setting a life cycle for the data detail table partitioned by processing time. In the actual application scenario of the embodiment of the present application, the application value of most of the data generated by the vehicle side will decrease with the passage of time. To this end, in order to maximize the application value of the generated data warehouse and reduce the storage cost of the data warehouse, the embodiment of the present application can set a life cycle for at least one of the first wide table data or the data detail table partitioned by processing time, and delete the corresponding wide table data or detail table when the life cycle is reached.

[0049] The present application provides a streaming data processing method, which parses the ciphertext data to be stored generated by the vehicle side into plaintext data to be stored, and pushes it to an intermediate Kafka stream data processing platform; uses the intermediate Kafka stream data processing platform to process the plaintext data to be stored to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that fully represents the vehicle-side event data; through the first wide table data, supports data calls of real-time data acquisition work scenarios of downstream business parties; and / or, stores the second wide table data in OSS object service storage to support data calls by downstream business parties. The Willow data processing method provided in the embodiment of the present application uses an intermediate Kafka stream data processing platform to process the data generated by the vehicle side, and then generates wide table data partitioned by event time for default data and non-default data, and generates a data warehouse to support the vehicle development needs of downstream business parties. The data warehousing process does not require the data warehouse generator to consider issues such as the complexity of the signal format of the warehoused data and data transmission delays. It can efficiently process and analyze massive amounts of data, realize timely warehousing of vehicle-side data, and facilitate downstream business parties to retrieve and use it.

[0050] Embodiment 2

[0051] Based on the streaming data processing method described in the first embodiment of the present application, the present application also provides a streaming data processing device 20, such as Figure 2 As shown, Figure 2 A structural diagram of a streaming data processing device 20 provided in an embodiment of the present application, the streaming data processing device 20 includes:

[0052] The push module 201 is used to parse the encrypted data to be stored generated by the vehicle end into plaintext data to be stored, and then push it to the intermediate Kafka stream data processing platform;

[0053] A generation module 202 is used to process the plaintext data to be stored by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that completely represents vehicle-side event data;

[0054] The storage module 203 is used to support the data call of the real-time data acquisition work scenario of the downstream business party through the first wide table data; and / or store the second wide table data in the OSS object service storage to support the downstream business party to make data calls.

[0055] Optionally, optionally, in one embodiment of the present application, the generation module 202 is also used to: when generating the first wide table data, use the intermediate kafka stream data processing platform to process the plaintext data to be stored, and generate a data list partitioned by processing time; based on the data list, execute an hourly scheduling strategy for non-delayed data, and / or execute a batch scheduling strategy for delayed data to generate the second wide table data.

[0056] Optionally, in an embodiment of the present application, the device 20 also includes a transfer module (not shown in the drawings), which is used to determine the called cold data whose utilization rate is lower than a preset rate from the generated second wide table data; and transfer the cold data to the OSS object storage service.

[0057] Optionally, in one embodiment of the present application, the push module 201 is also used to: convert the plaintext data to be stored into preset format data based on the necessary signal information determined by the data requirements of the downstream business; and push the format data obtained after the conversion to the intermediate kafka stream data processing platform. In this way, while ensuring that the data stored in the data warehouse meets the business development needs of the downstream business party, it also saves the required data warehouse space, and also facilitates the downstream business party to directly obtain data for use without further processing, reducing the work links of the downstream business party's use of data and improving the efficiency of data acquisition.

[0058] Optionally, in one embodiment of the present application, the device 20 also includes a full module (not shown in the drawings): the full module is used to store the encrypted data to be stored in the database to be stored in the OSS object storage service, so that downstream business demand parties can query and obtain the required data.

[0059] Optionally, in an embodiment of the present application, the device 20 also includes an interaction module (not shown in the drawings), which is used to generate a signal export dashboard corresponding to the encrypted data to be stored, so as to support downstream business demand parties to query the data stored in the OSS object storage service.

[0060] Optionally, in one embodiment of the present application, the generation module 202 is also used to use the intermediate kafka stream data processing platform to perform ETL aggregation operations on the plaintext data to be stored, obtain necessary data to be stored, and generate second wide table data corresponding to the necessary data to be stored. Correspondingly, the second wide table data is stored in the OSS object service storage to support the downstream business party to make data calls, including: statistics on the demand data of the downstream business party, and determine the real-time acquisition of indicator information or offline acquisition of statistical information; based on the real-time acquisition of indicator information or offline acquisition of statistical information, combined with the second wide table data stored in the OSS object service storage for calling.

[0061] Optionally, in one embodiment of the present application, the method further includes a deletion module (not shown in the accompanying drawings), which is used to set a preset life cycle for the first wide table data, and / or to set a life cycle for the data detail table partitioned by processing time.

[0062] The present application provides a streaming data processing device, which is provided with a push module to parse the ciphertext data to be stored generated by the vehicle side into plaintext data to be stored, and then push it to an intermediate Kafka stream data processing platform; a generation module is provided to process the plaintext data to be stored using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that completely represents the vehicle side event data; a storage module is provided to support data calls of real-time data acquisition work scenarios of downstream business parties through the first wide table data; and / or, the second wide table data is stored in the OSS object service storage to support data calls by downstream business parties. The willow data processing method provided in the embodiment of the present application uses the intermediate Kafka stream data processing platform to process the data generated by the vehicle end and then generates default data and non-default data partitioned by event time wide table data, generates a data warehouse to support the vehicle development needs of downstream business parties, and the data warehousing process does not require the data warehouse generation to consider the complexity of the signal format of the warehoused data and data transmission delay. It can efficiently process and analyze massive data, realize the timely warehousing of vehicle-end data, and facilitate downstream business parties to retrieve and use it. The structure of the device is simple and easy to implement and maintain.

[0063] Embodiment 3

[0064] The present application also provides a storage medium on which a computer program is stored. When the program is executed by a processor, any one of the streaming data processing methods described in the first embodiment of the present application is implemented. The streaming data processing method includes but is not limited to:

[0065] After parsing the encrypted data to be stored generated by the vehicle into plaintext data to be stored, it is pushed to the intermediate Kafka stream data processing platform;

[0066] The plaintext data to be stored is processed by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that completely represents vehicle-side event data;

[0067] The first wide table data is used to support data calls of real-time data acquisition work scenarios of downstream business parties; and / or the second wide table data is stored in OSS object service storage to support data calls by downstream business parties.

[0068] The present application also provides an electronic device, such as Figure 3 As shown, the electronic device includes:

[0069] One or more processors 301;

[0070] The memory 303 is used to store one or more programs.

[0071] When the one or more programs are instructed to be executed by the one or more processors 301 through the communication bus 304 based on the operation command received through the communication interface 302, the one or more processors 301 implement the streaming data processing method as described in any one of the above embodiments. The streaming data processing method includes but is not limited to:

[0072] After parsing the encrypted data to be stored generated by the vehicle into plaintext data to be stored, it is pushed to the intermediate Kafka stream data processing platform;

[0073] The plaintext data to be stored is processed by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that completely represents vehicle-side event data;

[0074] The first wide table data is used to support data calls of real-time data acquisition work scenarios of downstream business parties; and / or the second wide table data is stored in OSS object service storage to support data calls by downstream business parties.

[0075] So far, the application has described specific embodiments of the subject matter. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0076] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system layer on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0077] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0078] The system layers, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0079] For the convenience of description, the above device is described in terms of functions and is described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0080] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, system layers, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0081] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0082] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific transactions or implement specific abstract data types. The present application may also be practiced in distributed computing environments where transactions are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0083] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system layer embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0084] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A streaming data processing method, characterized in that: include: After parsing the encrypted data to be stored generated by the vehicle into plaintext data to be stored, it is pushed to the intermediate Kafka stream data processing platform; The plaintext data to be stored is processed by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that completely represents vehicle-side event data; The first wide table data is used to support data calls of real-time data acquisition work scenarios of downstream business parties; and / or the second wide table data is stored in OSS object service storage to support data calls by downstream business parties.

2. The streaming data processing method according to claim 1, characterized in that: The using the intermediate Kafka stream data processing platform to process the plaintext data to be stored to generate first wide table data and / or second wide table data includes: When generating the first wide table data, the intermediate Kafka stream data processing platform is used to process the plaintext data to be stored, and a data detail table partitioned by processing time is generated; Based on the data detail table, an hourly scheduling strategy is executed for non-delayed data, and / or a batch scheduling strategy is executed for delayed data to generate the second wide table data.

3. The streaming data processing method according to claim 2, characterized in that: The method further includes: determining, from the generated second wide table data, called cold data having a utilization rate lower than a preset rate; The cold data is transferred to the OSS object storage service.

4. The streaming data processing method according to claim 1, characterized in that: After parsing the ciphertext data to be stored into plaintext data to be stored, the data is pushed to the intermediate Kafka stream data processing platform, including: based on the necessary signal information determined by the data requirements of the downstream business, converting the plaintext data to be stored into preset format data; The format data obtained after conversion is pushed to the intermediate Kafka stream data processing platform.

5. The streaming data processing method according to claim 1, characterized in that: The method further includes: storing the encrypted data to be stored in the database to be stored in the OSS object storage service, so that the downstream business demand party can query and obtain the required data.

6. The streaming data processing method according to claim 5, characterized in that: The method further includes: generating a signal export dashboard corresponding to the encrypted data to be stored, so as to support downstream business demanders to query the data stored in the OSS object storage service.

7. The streaming data processing method according to claim 1, characterized in that: The using the intermediate Kafka stream data processing platform to process the plaintext data to be stored to generate first wide table data and / or second wide table data includes: Using the intermediate Kafka stream data processing platform to perform ETL aggregation operations on the plaintext data to be stored, and obtaining necessary data to be stored; generating second wide table data corresponding to the necessary data to be stored; Correspondingly, the storing of the second wide table data in the OSS object service storage to support the downstream business party to call data includes: performing statistics on the demand data of the downstream business party to determine whether to obtain indicator information in real time or obtain statistical information offline; Based on the real-time acquisition of indicator information or offline acquisition of statistical information, the call is made in combination with the second wide table data stored in the OSS object service storage.

8. The streaming data processing method according to claim 2, characterized in that: The method further includes: setting a preset life cycle for the first wide table data, and / or setting a life cycle for the data detail table partitioned by processing time.

9. A streaming data processing device, characterized in that: include: The push module is used to parse the encrypted data to be stored generated by the vehicle into plaintext data to be stored, and then push it to the intermediate Kafka stream data processing platform; A generation module, used to process the plaintext data to be stored by using the intermediate Kafka stream data processing platform to generate first wide table data and / or second wide table data, wherein the first wide table data and the second wide table data are both wide table data partitioned according to event time; wherein the first wide table data is wide table data with a rolling window set according to event time, and the second wide table data is wide table data that completely represents vehicle-side event data; The storage module is used to support the data call of the real-time data acquisition work scenario of the downstream business party through the first wide table data; and / or store the second wide table data in the OSS object service storage to support the downstream business party to make data calls.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and when the computer executable instructions are executed, the streaming data processing method according to any one of claims 1 to 8 is executed.