Data association system based on big data platform

Abstract

The invention discloses a data association system based on a big data platform, and belongs to the technical field of big data. Comprising a data acquisition module used for acquiring and preprocessing structured data and unstructured data of a vehicle to obtain processed data; the big data platform is connected with the data acquisition module and comprises a data lake used for storing the processed data; the batch and stream integrated calculation module is connected with the data lake and is used for performing batch processing and stream processing on the processed data to obtain calculated data; and the online analysis processing database is connected with the batch flow integrated calculation module and is used for associating and retrieving the calculated data. The technical scheme has the beneficial effects that the structured data and the unstructured data are stored based on the data lake technology, the structured data and the unstructured data are effectively associated by utilizing the big data platform, an efficient information retrieval service is provided, and comprehensive data utilization requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data association system based on a big data platform. Background Art

[0002] Currently, major OEMs have equipped their vehicles with a variety of cameras, LiDAR, millimeter-wave radar and other hardware devices, which collect massive amounts of unstructured data such as videos, images, point cloud files, etc. At the same time, sensors on vehicles also collect massive amounts of structured data, including vehicle status information and certain driver behaviors, such as acceleration, whether there is braking action, etc. OEMs collect this data to train intelligent driving models and provide other digital services to users.

[0003] Considering the data flow factor and the massive amount of data required for model training, the road collection car collects unstructured data every day, saves the data to the hard disk, and then uploads it to the cloud by mounting the hard disk on the server. After desensitization, cleaning and other step-by-step processing, high-quality materials are generated, and then handed over to the labeling personnel for repeated labeling, and finally transmitted to the intelligent driving model for training. The structured data in the road collection car is compressed and encoded on the vehicle side, and uploaded to the cloud through the vehicle side network at regular intervals. It is also desensitized, cleaned and other data processing, and is saved in the data warehouse, and is finally used to analyze the vehicle status and driving behavior.

[0004] In the existing technology, the structured data and unstructured data of road-tested vehicles and mass-produced vehicles are transmitted to the cloud through different channels, and stored and processed separately, resulting in no effective association between the data, making it difficult to fully understand the vehicle's condition, state changes, and driver operation behavior within a certain period of time. This situation makes the final data utilization inconvenient. Summary of the invention

[0005] The purpose of the present invention is to provide a data association system based on a big data platform to solve the above technical problems.

[0006] A data association system based on a big data platform, comprising:

[0007] The data acquisition module is used to collect the structured data and unstructured data of the vehicle and perform pre-processing to obtain processed data;

[0008] A big data platform is connected to the data acquisition module, and the big data platform includes:

[0009] A data lake, used to store the processed data;

[0010] A batch-stream integrated computing module is connected to the data lake, and performs batch processing and stream processing on the processed data to obtain calculated data;

[0011] The online analytical processing database is connected to the batch-stream integrated computing module and is used to associate and retrieve the calculated data.

[0012] Preferably, the data acquisition module includes:

[0013] A data collection unit, which collects the unstructured data through environmental perception sensors of the mass-produced vehicle and the road-produced vehicle, and collects the structured data through vehicle sensors of the mass-produced vehicle and the road-produced vehicle;

[0014] A data uploading unit, connected to the data collecting unit, used for compressing the unstructured data and the structured data and uploading them to a cloud server;

[0015] The preprocessing unit is connected to the data uploading unit, and performs decompression, compliance check, desensitization, video frame extraction and image deduplication processing on the unstructured data and the structured data in sequence to obtain the processed data.

[0016] Preferably, the data uploading unit includes:

[0017] A real-time uploading subunit uploads the structured data and the unstructured data collected by the mass-produced vehicle to the cloud server in real time via the network;

[0018] The offline uploading subunit uploads the unstructured data collected by the road collection vehicle to the cloud server through offline copying after the road collection is completed.

[0019] Preferably, the batch processing is to divide the unstructured data collected by the road sampling vehicle into multiple batches for offline analysis and data correction.

[0020] Preferably, the stream processing is to perform real-time analysis on the structured data and the unstructured data collected by the mass-produced vehicle.

[0021] Preferably, the data in the data lake is associated and matched by vehicle frame number and timestamp, and the accuracy of the timestamp is at the second level.

[0022] Preferably, the data delay of the structured data and the unstructured data collected by the mass-produced vehicle in the pre-processing unit is at the level of minutes.

[0023] Preferably, the data delay of the unstructured data collected by the road sampling vehicle in the preprocessing unit is at the level of days.

[0024] Preferably, it also includes a data application module, which is connected to the big data platform and uses the calculated data to perform model training and / or data analysis based on actual data requirements.

[0025] Preferably, the offline copying is to connect the hard disk on the road mining vehicle to the cloud server for transmission or upload it to the cloud server through a file transfer protocol.

[0026] The beneficial effects of the present invention are: storing structured data and unstructured data based on data lake technology, utilizing the big data platform to effectively associate structured data and unstructured data, providing efficient information retrieval services, and meeting comprehensive data utilization needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a structural schematic diagram of the data association system based on the big data platform of the present invention;

[0028] Figure 2 It is a connection block diagram of the data association system based on the big data platform of the present invention;

[0029] Figure 3 It is a schematic diagram of a data uploading unit of the present invention.

[0030] In the attached figure: 1. Data acquisition module; 11. Data collection unit; 11a. Production vehicle collection subunit; 11b. Structured data collection subunit; 11c. Road collection vehicle collection subunit; 12. Data upload unit; 121. Real-time upload subunit; 122. Offline upload subunit; 12a. Production vehicle real-time upload subunit; 12b. Structured data upload subunit; 12c. Road collection vehicle offline upload subunit; 13. Preprocessing unit; 13a. Production vehicle storage and processing subunit; 13b. Structured data storage and processing subunit; 13c. Road collection vehicle storage and processing subunit; 2. Big data platform; 21. Data lake; 22. Batch-stream integrated computing module; 22a. Stream processing unit; 22b. Batch processing unit; 23. Online analytical processing database; 3. Data application module; 3a. Data analysis unit; 3b. Model training unit. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0033] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0034] A data association system based on a big data platform, such as Figures 1 to 3 As shown, including,

[0035] The data collection module 1 is used to collect the structured data and unstructured data of the vehicle and perform preprocessing to obtain processed data;

[0036] The big data platform 2 is connected to the data collection module 1. The big data platform 2 includes:

[0037] Data lake 21, used to store processed data;

[0038] The batch-stream integrated computing module 22 is connected to the data lake 21, and performs batch processing and stream processing on the processed data to obtain the calculated data;

[0039] The online analytical processing database 23 is connected to the batch-stream integrated computing module 22 and is used to associate and retrieve the calculated data.

[0040] Specifically, the present invention provides a data association system based on a big data platform, which is mainly used in data collection and data management platforms. It can efficiently manage, calculate and associate massive amounts of data, use the big data platform 2 to associate structured data with unstructured data, provide efficient information retrieval services, and meet comprehensive data utilization needs.

[0041] Based on the data lake 21 technology, a large amount of data with various structures can be stored. The online analytical processing database 23 based on column storage can provide efficient information retrieval for the business side. At the same time, the accurate integration of structured data and unstructured data can more comprehensively reflect the vehicle status and external environment at a certain point in time or a certain period of time. Provide high-quality data for activities such as intelligent driving training and user driving habit analysis.

[0042] Unstructured data includes office documents of all formats, text, pictures, XML, HTML, various reports, images, and audio / video information, etc. In the present invention, it mainly refers to files such as images, audio / video, and point clouds.

[0043] Online analytical processing database 23, also known as OLAP database, is a database management system specially designed to support complex data analysis and decision support systems. Use data lake 21 to store structured and unstructured data, use Flink for offline and real-time computing, and use distributed OLAP database to provide efficient association and retrieval of business-side data.

[0044] In a preferred embodiment, the data acquisition module 1 includes:

[0045] A data collection unit 11 collects unstructured data through environmental perception sensors of mass-produced vehicles and road-produced vehicles, and collects structured data through vehicle sensors of mass-produced vehicles and road-produced vehicles;

[0046] A data uploading unit 12, connected to the data collecting unit 11, is used to compress the unstructured data and the structured data and upload them to the cloud server;

[0047] The preprocessing unit 13 is connected to the data uploading unit 12, and performs decompression, compliance check, desensitization, video frame extraction and image deduplication processing on the unstructured data and the structured data in sequence to obtain processed data.

[0048] Specifically, Figure 2 As shown, the mass-produced vehicle collection subunit 11a triggers the collection of unstructured data (such as images, videos, point cloud files) through the environmental perception sensors (including lidar, millimeter-wave radar, high-definition camera and ultrasonic radar) of the mass-produced vehicle when certain events occur (such as emergency braking) or at a certain time interval (such as 2 minutes).

[0049] The structured data collection subunit 11b triggers the collection of structured data (such as battery power) through the vehicle sensors of mass-produced vehicles and road-tested vehicles when certain events occur (such as low battery power) or at a certain time interval (such as 15 seconds).

[0050] The road collection vehicle collection subunit 11c is a road collection vehicle that uses the same environmental perception sensors as mass-produced vehicles to collect data. The road collection vehicle drives on normal roads for a long time, collects a large amount of unstructured data, and saves it to the hard disk of a dedicated device on the vehicle.

[0051] Specifically, through the environmental perception sensors and vehicle sensors of mass-produced vehicles and road vehicles, the data collection module 1 can collect unstructured data and structured data at the same time to ensure the comprehensiveness and diversity of the data. The sensors directly collect data, reducing the possibility of human intervention and improving the real-time and accuracy of the data.

[0052] The data uploading unit 12 compresses the unstructured data and the structured data, thereby reducing the bandwidth and time required for data transmission and improving the upload efficiency. The data is uploaded to the cloud server, and the complexity and possible errors of data transmission are reduced through compression, thereby enhancing the stability of data upload.

[0053] In the preprocessing unit 13, the decompression operation restores the compressed data to ensure that the original data can be used for subsequent processing and analysis. Among them, the data is checked for compliance to ensure that the data complies with relevant laws, regulations and industry standards, reducing legal risks in data processing. Sensitive information is desensitized to protect user privacy and data security. Through frame extraction and deduplication processing, the storage and analysis of video and image data are optimized, redundant data is reduced, and the efficiency of data processing is improved.

[0054] In a preferred embodiment, referring to Figure 2 , Figure 3 , the data uploading unit 12 includes,

[0055] The real-time uploading subunit 121 uploads the structured data and the unstructured data collected by the mass-produced vehicle to the cloud server in real time via the network;

[0056] The offline uploading subunit 122 uploads the unstructured data collected by the road collection vehicle to the cloud server through offline copying after the road collection is completed.

[0057] Specifically, the real-time uploading subunit 12a of the mass-produced vehicle uploads the data to the cloud in real time through the network after data compression; the structured data uploading subunit 12b uploads the data to the cloud in real time through the network after data compression; the offline uploading subunit 12c of the road procurement vehicle uploads the data to the cloud by offline copying after the road procurement is completed.

[0058] In a preferred embodiment, batch processing is to divide the unstructured data collected by the road collection vehicle into multiple batches for offline analysis and data correction;

[0059] Stream processing is the real-time analysis of structured data and unstructured data collected by mass-produced vehicles.

[0060] Specifically, the matching of real-time data of the present invention can be achieved with a delay of minutes. However, data inaccuracy may occur, such as delayed uploading of unstructured data. The advantages of batch processing and stream processing are combined. Batch processing unit 22b is usually used to process large-scale historical data, while stream processing unit 22a is used for real-time data streams. Batch-stream integrated computing can process historical data and real-time data simultaneously in the same system, thereby achieving more efficient analysis and response.

[0061] Offline data matching still has a delay of days. At the same time, the batch processing program will make corrections to data matching and summary statistics to ensure the final correctness.

[0062] In a preferred embodiment, the data in the data lake 21 is associated and matched by the vehicle frame number and the timestamp, and the accuracy of the timestamp is at the second level.

[0063] Specifically, all unstructured data and structured data are still aggregated and saved in the data lake 21 after processing. Association matching is performed through the vehicle frame number and timestamp. Since the data comes from different vehicle-side sensor modules, range matching can be achieved within seconds. At this time, there may be data that is not matched, but this data still has business value, so it continues to be retained in the data lake 21.

[0064] In a preferred embodiment, the data delay of structured data and unstructured data collected by mass-produced vehicles in the pre-processing unit 13 is at the level of minutes;

[0065] The data delay of the unstructured data collected by the road collection vehicle in the pre-processing unit 13 is at the level of days.

[0066] Specifically, the unstructured data of the mass-produced vehicles of the mass-produced vehicle storage and processing subunit a is saved in the data lake 21, and then checked and processed, including decompression, compliance check, desensitization, video frame extraction, image deduplication, improving data quality and reducing data quantity. The above process is carried out in real time, and the delay is controlled at the minute level.

[0067] The structured data of the mass-produced vehicles and road-produced vehicles of the structured data storage and processing subunit 13b are stored in the data lake 21, and then decoded, desensitized, and deduplicated. The data delay is controlled at the minute level.

[0068] The unstructured data of the road collection vehicle storage and processing subunit 13c is saved in the data lake 21, and is also subjected to offline batch decompression, compliance check, desensitization, video frame extraction, and image deduplication. The delay of the entire data processing is at the day level.

[0069] In a preferred embodiment, it also includes a data application module 3, which is connected to the big data platform 2 and uses the calculated data to perform model training and / or data analysis based on actual data requirements.

[0070] Specifically, the data analysis unit 3b can select real-time data or offline data to perform model training according to actual data requirements. The data analysis unit 3a can use real-time data or offline data to perform data statistics and analysis according to actual data requirements.

[0071] The connection between the data application module 3 and the big data platform 2 promotes the flow and integration of data between different systems, and realizes the seamless application and optimization of data.

[0072] The processed data and trained models can be applied to multiple scenarios, such as intelligent prediction, personalized recommendation, business optimization, etc., expanding the application scope and value of the data.

[0073] In a preferred embodiment, offline copying is to connect the hard disk on the road mining vehicle to the cloud server for transmission or upload it to the cloud server via the file transfer protocol (FTP).

[0074] Specifically, offline replication can improve the efficiency and reliability of data transmission by directly connecting the hard disk to the cloud server or uploading it to the cloud server using FTP. It avoids the impact of network delays and interruptions, ensures the integrity and accuracy of the data, and saves network bandwidth resources.

[0075] The present invention can improve the comprehensiveness and accuracy of data collection, optimize the efficiency and stability of data uploading, ensure the compliance and security of data processing, and at the same time improve the efficiency and quality of data preprocessing, which helps to improve the overall effect of data processing and analysis and support subsequent data mining and application.

[0076] Based on data lake technology, massive amounts of data with various structures can be stored. Using a distributed system parallel computing system, massive amounts of data can be efficiently managed, calculated, and associated. At the same time, the OLAP database based on column storage can provide efficient information retrieval for the business side. At the same time, the accurate integration of structured data and unstructured data can more comprehensively reflect the vehicle status and external environment at a certain point in time or within a certain period of time. Provide high-quality data for activities such as intelligent driving training and user driving habit analysis.

[0077] With the increase of vehicle-cloud bandwidth and the development of data compression technology, the offline batch data collection method of road collection vehicles can be completely replaced by real-time unstructured data collection of mass-produced vehicles. By then, the data utilization delay on the business side can be reduced to minutes, and the quality of the all-round data view will be greatly improved, which will greatly meet the business side's demand for massive high-quality data.

[0078] In summary, the present application provides a data association system based on a big data platform, which associates structured data and unstructured data through the big data platform 2 in the cloud to build a complete data view to meet the needs of all-round data utilization. Among them, the real-time data delay is controlled at the minute level to meet the real-time requirements of data utilization. At the same time, the offline data delay is controlled at the day level to meet the requirements of data accuracy.

[0079] The above description is only a preferred embodiment of the present invention, and does not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A data association system based on a big data platform, characterized in that: include, A data collection module (1) is used to collect structured data and unstructured data of the vehicle and perform preprocessing to obtain processed data; A big data platform (2) is connected to the data acquisition module (1), and the big data platform (2) includes: A data lake (21), used for storing the processed data; A batch-stream integrated computing module (22), connected to the data lake (21), performs batch processing and stream processing on the processed data to obtain computed data; An online analytical processing database (23) is connected to the batch-stream integrated computing module (22) and is used to associate and retrieve the calculated data.

2. The data association system based on the big data platform according to claim 1 is characterized in that: The data acquisition module (1) comprises: A data collection unit (11) collects the unstructured data through environmental perception sensors of a mass-produced vehicle and a road-produced vehicle, and collects the structured data through vehicle sensors of the mass-produced vehicle and the road-produced vehicle; A data uploading unit (12), connected to the data collecting unit (11), used for compressing the unstructured data and the structured data and uploading them to a cloud server; The preprocessing unit (13) is connected to the data uploading unit (12) and performs decompression, compliance check, desensitization, video frame extraction and image deduplication processing on the unstructured data and the structured data in sequence to obtain the processed data.

3. The data association system based on the big data platform according to claim 2 is characterized in that: The data uploading unit (12) comprises: A real-time uploading subunit (121) uploads the structured data and the unstructured data collected by the mass-produced vehicle to the cloud server in real time via a network; The offline uploading subunit (122) uploads the unstructured data collected by the road collection vehicle to the cloud server through offline copying after the road collection is completed.

4. The data association system based on the big data platform according to claim 2 is characterized in that: The batch processing is to divide the unstructured data collected by the road sampling vehicle into multiple batches for offline analysis and data correction.

5. The data association system based on the big data platform according to claim 2 is characterized in that: The stream processing is to perform real-time analysis on the structured data and the unstructured data collected by the mass-produced vehicle.

6. The data association system based on the big data platform according to claim 1 is characterized in that: The data in the data lake (21) is associated and matched by the vehicle frame number and the timestamp, and the accuracy of the timestamp is at the second level.

7. The data association system based on the big data platform according to claim 2 is characterized in that: The data delay of the structured data and the unstructured data collected by the mass-produced vehicle in the pre-processing unit (13) is at the level of minutes.

8. The data association system based on the big data platform according to claim 2 is characterized in that: The data delay of the unstructured data collected by the road sampling vehicle in the preprocessing unit (13) is at the level of days.

9. The data association system based on the big data platform according to claim 1, characterized in that: It also includes a data application module (3) connected to the big data platform (2) and using the calculated data to perform model training and / or data analysis based on actual data requirements.

10. The data association system based on the big data platform according to claim 3, characterized in that: The offline copying is to connect the hard disk on the road mining vehicle to the cloud server for transmission or upload it to the cloud server through a file transfer protocol.

Images