Method, system and equipment for integrating multi-source data of test-taking and test-driving scene

Abstract

The invention relates to the technical field of big data processing and analysis, and discloses a method, a system and equipment for integrating multi-source data of a test-taking and test-driving scene. The method comprises the following steps: capturing original test-taking and test-driving data from a plurality of HBase tables in real time based on an Flink technology; performing cleaning, conversion and integration on the original test driving data based on a user-defined function and conversion operation of Flink to obtain target test driving data; storing structured metadata and index data in the target trial-riding and trial-driving data into a preset unified database; storing the video file to a distributed file system, and storing preset key data to a memory database; and performing test driving evaluation and optimization based on the stored target test driving data. According to the method, the real-time stream processing capability of the Flink is combined with the optimized data reading, conversion and storage strategies, so that the multi-source data can be quickly, accurately and consistently integrated, and support is provided for the decision-making of the test-driving and test-driving business.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing and analysis, and in particular to a method, system and device for integrating multi-source data of a test ride and test drive scenario. Background Art

[0002] With the development of the automobile industry, test rides have become an indispensable part of the automobile sales process. In order to improve the efficiency of test ride services and customer experience, major automobile companies have introduced big data analysis technology to optimize processes and improve service quality. However, in the test ride scenario, the data involved is often scattered and stored in multiple HBase tables. These tables may be maintained by different systems, with different data formats and a lot of redundancy and duplication. Traditional data integration methods are mostly based on batch processing frameworks such as Hadoop, which is difficult to meet the real-time and efficiency requirements of the test ride business. In the test ride business, it is urgent to solve the problems of low efficiency of multi-scenario value acquisition and difficulty in ensuring data consistency due to diverse data sources, large data volume, and high real-time requirements. Summary of the invention

[0003] The main purpose of the present invention is to provide a method, system and device for integrating multi-source data in a test ride and test drive scenario, aiming to solve at least one of the above-mentioned technical problems.

[0004] To achieve the above object, the present invention provides a method for integrating multi-source data of a test ride and test drive scenario, comprising:

[0005] Capture raw test drive data from multiple HBase tables in real time based on Flink technology;

[0006] Clean, transform and integrate the original test drive data based on Flink’s custom functions and transformation operations to obtain target test drive data;

[0007] Storing structured metadata and index data in the target test drive data into a preset unified database;

[0008] Storing the video file of the target test drive data in a distributed file system, and storing the preset key data of the target test drive data in a memory database;

[0009] Test drive evaluation and optimization are performed based on the stored target test drive data.

[0010] In some embodiments, the real-time capture of raw test ride data from multiple HBase tables based on Flink technology includes:

[0011] Evenly divide the data into multiple Flink tasks according to the partitions of the HBase table; wherein each of the Flink tasks reads data from one or more partitions;

[0012] Dynamically adjust the number and resource allocation of the Flink tasks through a load balancing algorithm to determine the partition that each Flink task corresponds to;

[0013] Based on the Flink task, multiple HBase tables are read in parallel to obtain the original test drive data.

[0014] In some embodiments, the Flink-based custom function and conversion operation cleans, converts, and integrates the original test drive data to obtain target test drive data, including:

[0015] The original test drive data is filtered and corrected based on the custom function of Flink to obtain the cleaned test drive data;

[0016] Performing data standardization processing on the cleaned test drive data based on the conversion operation of Flink to obtain standardized test drive data;

[0017] The standardized test ride and test drive data are integrated to obtain target test ride and test drive data.

[0018] In some embodiments, the Flink-based custom function filters and modifies the original test drive data to obtain cleaned test drive data, including:

[0019] Identify and delete duplicate records of the original test drive data according to a unique index function in the database or write an SQL query statement to obtain cleaned data;

[0020] Setting a threshold range, identifying the cleaned data based on the threshold range and correcting erroneous data to obtain corrected data;

[0021] The corrected data is screened according to the query start and end time of the test drive and the frame number to retain the test drive analysis data and obtain the cleaned test drive data.

[0022] In some embodiments, the Flink-based conversion operation performs data standardization processing on the cleaned test drive data to obtain standardized test drive data, including:

[0023] Obtaining the timestamp of the cleaned test ride data;

[0024] The timestamp is converted into a unified format based on a conversion operation of Flink to obtain first converted data;

[0025] Obtaining speed and acceleration units of the first conversion data;

[0026] The speed and acceleration units are standardized based on the conversion operation of Flink to obtain second conversion data;

[0027] The data format of the second conversion data is standardized based on the conversion operation of Flink to obtain standardized test ride and test drive data.

[0028] In some embodiments, the step of integrating the standardized test drive data to obtain target test drive data includes:

[0029] Performing trajectory data integration on the standardized test ride and test drive data to obtain first integrated data;

[0030] Acquire the ignition and shutdown status data of the first integrated data;

[0031] The ignition on / off status data is associated and integrated with the timestamp, speed and acceleration data to obtain target test ride and test drive data.

[0032] In some embodiments, the test drive evaluation and optimization based on the stored target test drive data includes:

[0033] Create indexes for key fields;

[0034] When receiving a data query instruction, the statement of the data query instruction is automatically optimized based on the SQL query optimizer to obtain an optimized query statement;

[0035] Querying the stored target test drive data according to the optimized query statement and the index, obtaining a query result and caching the query result;

[0036] Test drive evaluation and optimization are performed based on the query results.

[0037] In some embodiments, the method further comprises:

[0038] Build a message queue as data middleware;

[0039] When a test drive vehicle generates warning information, receiving the warning information based on the message queue;

[0040] Obtain the warning information in real time by subscribing to the messages in the message queue;

[0041] Based on the warning information, an asynchronous processing method is used to trigger a warning video generation process to obtain a video data stream;

[0042] Build a real-time stream processing framework;

[0043] The video data stream is analyzed and processed in real time based on the real-time stream processing framework to generate corresponding warning videos or alarm information.

[0044] In addition, to achieve the above-mentioned purpose, the present invention also proposes a test drive scenario multi-source data integration system, comprising:

[0045] Data capture module, which is used to capture raw test drive data from multiple HBase tables in real time based on Flink technology;

[0046] A data conversion module is used to clean, convert and integrate the original test drive data based on Flink's custom functions and conversion operations to obtain target test drive data;

[0047] A first data storage module is used to store structured metadata and index data in the target test drive data into a preset unified database;

[0048] A second data storage module is used to store the video file of the target test drive data in a distributed file system, and store the preset key data of the target test drive data in a memory database;

[0049] A data analysis module is used to perform test drive evaluation and optimization based on the stored target test drive data.

[0050] In addition, to achieve the above-mentioned purpose, the present invention also proposes an electronic device, which includes: a memory, a processor, and a multi-source data integration program for test ride and test drive scenarios stored on the memory and executable on the processor, wherein the multi-source data integration program for test ride and test drive scenarios is configured to implement the multi-source data integration method for test ride and test drive scenarios as described above.

[0051] The present invention provides a method for integrating multi-source data in a test drive scenario, including: capturing original test drive data from multiple HBase tables in real time based on Flink technology; cleaning, converting and integrating the original test drive data based on Flink's custom functions and conversion operations to obtain target test drive data; storing structured metadata and index data in the target test drive data in a preset unified database; storing the video file of the target test drive data in a distributed file system, and storing the preset key data of the target test drive data in a memory database; and performing test drive evaluation and optimization based on the stored target test drive data. In the present invention, multi-source data of test drive scenarios are efficiently integrated based on Flink big data processing technology. Through Flink's real-time stream processing capabilities combined with optimized data reading, conversion and storage strategies, fast, accurate and highly consistent integration of multi-source data is achieved, thereby providing strong support for decision-making in test drive business. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention;

[0053] Figure 2 It is a flowchart of an embodiment of a method for integrating multi-source data in a test ride and test drive scenario according to the present invention;

[0054] Figure 3 A schematic diagram of the system architecture hierarchy involved in the embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a specific example process involved in an embodiment of the present invention;

[0056] Figure 5 This is a structural block diagram of an embodiment of a multi-source data integration system for test ride and test drive scenarios of the present invention.

[0057] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0060] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0061] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present invention.

[0062] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM memory) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0063] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0064] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a test drive scenario multi-source data integration program.

[0065] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device, and the electronic device calls the test drive scene multi-source data integration program stored in the memory 1005 through the processor 1001, and executes the test drive scene multi-source data integration method provided in an embodiment of the present invention.

[0066] The present invention provides a method, system and device for integrating multi-source data in a test ride and test drive scenario.

[0067] The embodiment of the present invention provides a method for integrating multi-source data of a test drive scenario, referring to Figure 2 , Figure 2 It is a flowchart of an embodiment of a method for integrating multi-source data in a test ride and test drive scenario according to the present invention.

[0068] like Figure 2 As shown, the multi-source data integration method for test ride and test drive scenarios includes:

[0069] Step S100: capturing original test ride data from multiple HBase tables in real time based on Flink technology;

[0070] Step S200: Cleaning, converting and integrating the original test drive data based on Flink's custom functions and conversion operations to obtain target test drive data;

[0071] Step S300: storing structured metadata and index data in the target test drive data into a preset unified database;

[0072] Step S400: storing the video file of the target test drive data in a distributed file system, and storing the preset key data of the target test drive data in a memory database;

[0073] Step S500: performing test drive evaluation and optimization based on the stored target test drive data.

[0074] It should be noted that the execution subject in this embodiment may be an electronic device, which may be a computer device with data processing functions, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, a computer device is used as an example for explanation.

[0075] It is understandable that this embodiment uses the method based on Flink big data processing technology to efficiently integrate multi-source HBase (non-relational distributed database for column-based storage) data in the test drive scenario as an example for explanation. In the test drive scenario, the data involved is often scattered and stored in multiple HBase tables. These HBase tables are maintained by different systems, with different data formats and a large amount of redundancy and duplication. The method described in this embodiment is used to efficiently integrate multi-source HBase data in the test drive scenario, which solves the problems of low efficiency of multi-scenario value acquisition and difficulty in ensuring data consistency caused by diverse data sources, large data volume, and high real-time requirements in the test drive business.

[0076] For example, the method described in this embodiment can be implemented based on a multi-source data integration system for test ride and test drive scenarios, such as Figure 3 As shown in the figure, the system architecture mainly includes four parts: data collection layer, data processing layer, data storage layer and data application layer. Among them, the data collection layer is responsible for real-time capture of test drive related data from multiple HBase tables; the data processing layer uses Flink to clean, convert and integrate data; the data storage layer stores the processed data in a unified database for subsequent analysis; the data application layer provides intelligent decision support for test drive business based on the integrated data.

[0077] In one embodiment, raw test drive data are captured in real time from multiple HBase tables based on Flink technology, including: evenly dividing the data to multiple Flink tasks according to the partitions of the HBase table; wherein each of the Flink tasks reads data from one or more partitions; dynamically adjusting the number and resource allocation of the Flink tasks through a load balancing algorithm to determine the partitions that each Flink task corresponds to; and reading multiple HBase tables in parallel based on the Flink tasks to obtain raw test drive data.

[0078] For example, for data reading of HBase tables in a test drive scenario, the data collection layer of this embodiment adopts the following optimization strategies: parallel reading, caching strategy, and pre-reading technology.

[0079] Specifically, parallel reading: According to the partitioning of the HBase table, multiple Flink tasks are assigned to read data in parallel to improve data reading efficiency. Implementation process: ① Partitioning and task allocation: First, according to the partitioning of the HBase table, the data is evenly distributed to different Flink tasks. Each task is responsible for reading the data of one or more partitions, which can fully utilize the parallel computing capabilities of multi-core processors while avoiding the overhead of data transmission and replication. ② Load balancing: By dynamically adjusting the number of tasks and resource allocation, ensure that all tasks can fully utilize resources to avoid idle resources or excessive load. Using load balancing algorithms (such as polling, random selection, etc.) to determine which tasks should process which data partitions can further optimize the performance of the entire system. ③ Fault-tolerant mechanism: In parallel reading, the failure of a single task will not affect the execution of other tasks. By configuring reasonable error handling logic, such as retry mechanism, task recovery, etc., data consistency and integrity can be ensured.

[0080] In one embodiment, the method also includes: extracting hot data and commonly used data from HBase table data; caching the hot data and the commonly used data according to a preset cache mechanism based on a cache server cluster or a distributed cache solution to obtain cached data; updating the cached data regularly based on a preset time or according to a data access mode to obtain updated data so as to read the HBase table.

[0081] Specifically, cache strategy: For hot data and frequently used data, the use of a cache mechanism can reduce the access pressure of HBase and increase the data reading speed. Implementation process: ① Data type identification: For different types of queries (such as high-frequency queries, low-frequency queries), select appropriate cache strategies. For example, for frequently accessed hot data, you can use a more compact cache structure and set a shorter expiration time. ② Cache update strategy: Update the cache content regularly or according to the data access pattern to ensure that the data in the cache is always the latest. At the same time, design a reasonable cache expiration strategy to avoid long-term expiration of data in the cache. ③ Cache consistency: In a multi-node environment, it is necessary to solve the cache consistency problem, such as using a cache server cluster, a distributed cache solution (including but not limited to Redis, Memcached, etc.) to ensure data consistency and high availability.

[0082] In one embodiment, the method further includes: performing demand forecasting analysis based on a machine learning model according to historical data access patterns, user behaviors, and business indicators to obtain forecasting results; determining demand forecasting data based on the forecasting results; and pre-loading the demand forecasting data into a cache or fast access storage to capture original test ride data in real time.

[0083] Specifically, pre-reading technology: predicts and reads in advance the data that will be needed soon to ensure the real-time availability of the data. Implementation of pre-reading technology: ① Demand prediction: By analyzing historical data access patterns, user behavior, business indicators, etc., predict the data that may be needed in the future. In this embodiment, the machine learning model can be combined for prediction to improve the accuracy of the prediction. ② Data preparation: Based on the prediction results, pre-load the data into the cache or other fast-access storage. This step needs to consider factors such as storage costs and data update frequency to balance performance and resource utilization. ③ Dynamic adjustment: As the data access pattern changes, the pre-reading strategy also needs to be dynamically adjusted. By monitoring system performance indicators such as read latency, cache hit rate, etc., the timing and scope of pre-reading can be discovered and optimized in a timely manner.

[0084] In this embodiment, Flink technology is used for real-time stream processing. With Flink's real-time stream processing capabilities, data streams from multiple HBase tables can be read in real time. Flink can process large-scale data streams while ensuring low latency through its efficient parallel processing mechanism, meeting the real-time requirements of the test drive business.

[0085] In one embodiment, the original test drive data are cleaned, converted and integrated based on Flink's custom functions and conversion operations to obtain target test drive data, including: filtering and correcting the original test drive data based on Flink's custom functions to obtain cleaned test drive data; performing data standardization on the cleaned test drive data based on Flink's conversion operations to obtain standardized test drive data; and performing data integration on the standardized test drive data to obtain target test drive data.

[0086] Exemplarily, data conversion and integration: In the data processing layer, this embodiment uses Flink's user-defined functions (UDFs) and transformation operations to clean, convert and integrate data from different HBase tables to obtain target test drive data, including data cleaning, data standardization and data integration.

[0087] In one example, the original test drive data is filtered and corrected based on a custom function of Flink to obtain cleaned test drive data, including: identifying and deleting duplicate records of the original test drive data according to a unique index function in a database or writing an SQL query statement to obtain cleaned data; setting a threshold range, identifying the cleaned data based on the threshold range and correcting erroneous data to obtain corrected data; filtering the corrected data according to the query start and end time and frame number of the test drive to retain the test drive analysis data to obtain cleaned test drive data.

[0088] It is understandable that data cleaning: In the context of test drive data analysis, data cleaning is the primary step to ensure the accuracy and effectiveness of the analysis. Test drive data usually contain a large amount of real-time information related to vehicle driving, such as timestamps, speed, acceleration, position coordinates, ignition and shutdown status, etc. These data may be affected by various factors during the collection and transmission process, resulting in duplicate, erroneous or irrelevant records. Therefore, the effective cleaning of the original test drive data in this embodiment is the key to ensure that subsequent analysis can proceed smoothly. Data cleaning includes but is not limited to removing duplicate data, identifying and correcting erroneous data, and removing irrelevant data.

[0089] Specifically, duplicate data removal: Duplicate records need to be identified and deleted. In the test drive database, there may be duplicate data due to system errors or network delays. These data not only increase the complexity of data processing, but may also interfere with the analysis results. In order to remove duplicate data, the unique index function in the database can be used or SQL query statements can be written to find and delete duplicate records. In actual applications, during specific operations, key fields such as vehicle frame number (VIN) and timestamp can be compared to delete identical records.

[0090] Specifically, identify and correct erroneous data: erroneous data may include unreasonable speed values, timestamp errors, acceleration anomalies, etc. These errors may be caused by sensor failure, data transmission errors, or human input errors. In order to identify these erroneous data, a reasonable threshold range can be set, and data that exceeds the range (threshold range) can be marked and checked. For example, for speed data, a maximum speed threshold can be set, and speed records that exceed the threshold can be manually reviewed or automatically corrected.

[0091] Specifically, remove irrelevant data: data irrelevant to the test drive analysis needs to be removed. Test drive data may contain various information during the vehicle's driving process, but not all information is directly related to the test drive analysis. For example, the driving data of the vehicle during the non-test drive period and the driving data of non-test drive vehicles are all irrelevant data. In order to remove these irrelevant data, you can filter according to the query start and end time of the test drive and the frame number and other conditions, and only retain the data related to the test drive analysis.

[0092] In one example, the cleaned test drive data is subjected to data standardization based on a Flink conversion operation to obtain standardized test drive data, including: obtaining a timestamp of the cleaned test drive data; converting the timestamp into a unified format based on a Flink conversion operation to obtain first conversion data; obtaining speed and acceleration units of the first conversion data; standardizing the speed and acceleration units based on a Flink conversion operation to obtain second conversion data; and standardizing the data format of the second conversion data based on a Flink conversion operation to obtain standardized test drive data.

[0093] It should be noted that data standardization is the process of converting raw data into a unified format and unit to facilitate subsequent data analysis and processing. In the test drive data analysis, due to the diversity of data sources, the data may have different formats and units, such as timestamp format, speed unit, acceleration unit, etc. These differences will increase the complexity of data processing and reduce the efficiency of analysis. Therefore, standardizing the data is an important step to ensure the accuracy and consistency of the analysis. Data standardization includes but is not limited to timestamp standardization, speed and acceleration unit standardization, and data format standardization.

[0094] Specifically, timestamp standardization: Timestamps need to be standardized. Timestamps in test drive data may have different formats, such as second timestamps, millisecond timestamps, or string timestamps in custom formats. In order to unify the processing, all timestamps can be converted to a unified format, such as the unified time format of "yyyy-MM-dd HH:mm:ss", which is convenient for filtering and comparing time ranges and ensuring the accuracy of subsequent analysis results.

[0095] Specifically, speed and acceleration unit standardization: Since different vehicles or different sensors may use different speed units (such as kilometers per hour, meters per second) and acceleration units (such as meters per second 2, g, etc.), these units need to be uniformly converted. For example, all speed units can be uniformly converted to kilometers per hour, and acceleration units can be uniformly converted to meters per second 2, so as to eliminate the impact of unit differences on the analysis results and ensure the accuracy of subsequent analysis results.

[0096] Specifically, data format standardization: standardize the data format. For example, convert text-type data (such as date, time, address, etc.) into a unified format; convert numeric-type data (such as speed, acceleration, etc.) into a unified precision and type; convert Boolean-type data (such as ignition and shutdown status) into a unified representation (such as 0 for shutdown and 1 for ignition), thereby reducing errors and uncertainties in the data processing process and improving analysis efficiency.

[0097] In one example, data integration is performed on the standardized test ride and test drive data to obtain target test ride and test drive data, including: performing trajectory data integration on the standardized test ride and test drive data to obtain first integrated data; obtaining ignition and shutdown status data of the first integrated data; and associating and integrating the ignition and shutdown status data with timestamp, speed, and acceleration data to obtain target test ride and test drive data.

[0098] It is understandable that data integration is the process of combining and reconstructing cleaned and standardized data according to business logic to form a complete data view. In the test drive data analysis, data integration is an important basis for building analysis models, conducting data mining and predictive analysis. In this embodiment, through data integration, relevant information scattered in various data sources can be summarized and associated to provide comprehensive and accurate data support for subsequent analysis work. Data integration includes but is not limited to trajectory data integration and ignition and shutdown status integration.

[0099] Specifically, trajectory data integration: trajectory data is an important part of test drive analysis, which records the position changes and time information of the vehicle during driving. In order to form a complete trajectory view, the trajectory data in different time periods need to be sorted and merged in chronological order; at the same time, the trajectory data needs to be interpolated and smoothed to eliminate the impact of missing data and noise interference.

[0100] Specifically, ignition on / off status integration: The ignition on / off status is one of the important indicators reflecting the driving status of the vehicle. In order to accurately analyze the driving status changes during the test drive, it is necessary to associate and integrate the ignition on / off status data with timestamp, speed, acceleration and other data. In practical applications, by comparing the ignition on / off status data with timestamp data, etc., the correct ignition on / off status can be identified for subsequent analysis.

[0101] In one embodiment, the structured metadata and index data in the target test drive data are stored in a preset unified database; the video file of the target test drive data is stored in a distributed file system, and the preset key data of the target test drive data is stored in a memory database.

[0102] It should be noted that the data storage layer stores the processed data in a unified database for subsequent analysis. Specifically, data storage and query: In the autonomous driving vehicle warning video generation system, data storage and query are crucial links. Not only is it necessary to efficiently and stably store massive video data and its related metadata, but it is also necessary to be able to quickly respond to query requests and support supervisors to view and analyze warning videos in real time. The autonomous driving vehicle warning video generation system can be a subsystem in an electronic device (computer device). The following will explain in detail the three aspects of data storage solutions, data query optimization, and real-time query processing.

[0103] Specifically, the data storage solutions include: ① Unified database storage: The integrated data is stored in a unified database (such as Hive and MySQL). Hive, as a data warehouse tool based on Hadoop, is suitable for processing large-scale data sets and supports complex SQL queries. It can also store data on HDFS (Hadoop Distributed File System), making full use of Hadoop's distributed storage capabilities. MySQL, as a traditional relational database management system, is suitable for storing structured metadata and index data, such as vehicle information, warning type, and alarm time. ② Distributed file storage: In addition to database storage, video files themselves are more suitable for storage on distributed file systems (such as HDFS) due to their large size and frequent reading and writing. HDFS provides solid support for the storage of video data through its high fault tolerance, high throughput, and strong scalability. In HDFS, video files are cut into multiple data blocks and distributed on different nodes, which not only ensures data security but also improves the parallelism of data access. ③In-memory database acceleration: In this embodiment, in order to meet the needs of real-time queries, some key data is stored in memory databases such as Redis. Redis, with its high-performance key-value storage and rich data structure support, can quickly respond to query requests and greatly shorten the time for data retrieval. By caching frequently queried data in Redis, the number of database queries can be significantly reduced, the database load can be reduced, and the overall response speed can be improved.

[0104] In one embodiment, test drive evaluation and optimization are performed based on the stored target test drive data, including: indexing key fields; when receiving a data query instruction, automatically optimizing the statement of the data query instruction based on a SQL query optimizer to obtain an optimized query statement; querying the stored target test drive data according to the optimized query statement and the index to obtain a query result and caching the query result; and performing test drive evaluation and optimization based on the query result.

[0105] Specifically, data query optimization includes: ① Index optimization: In the database, index is an important means to improve query efficiency. By establishing indexes for key fields (such as vehicle number, alarm time, alarm type, etc.), the data retrieval speed can be accelerated. At the same time, according to the characteristics of the query mode, you can select the appropriate index type (such as B-tree index, hash index, etc.) and index strategy (such as single column index, composite index, etc.) to achieve the best query performance. ② Query statement optimization: Optimizing query statements is also the key to improving query efficiency. By avoiding full table scans, using effective connection conditions, reducing the nesting level of subqueries, and reasonably using aggregate functions and grouping operations, the complexity of query statements can be significantly reduced and query efficiency can be improved. In addition, the SQL query optimizer can be used to automatically optimize query statements and select the optimal execution plan based on data distribution and query mode. ③ Cache strategy: Cache is another effective means to improve data query performance. By caching query results, when the same data is queried again, the results can be directly obtained from the cache without accessing the database or file system again. In the cache strategy, it is necessary to reasonably set parameters such as cache validity period, elimination strategy and cache capacity to ensure the cache hit rate and real-time performance of data.

[0106] In one embodiment, the method also includes: constructing a message queue as data middleware; when a test drive vehicle generates warning information, receiving the warning information based on the message queue; obtaining the warning information in real time by subscribing to messages in the message queue; triggering a warning video generation process based on the warning information using an asynchronous processing method to obtain a video data stream; constructing a real-time stream processing framework; and performing real-time analysis and processing on the video data stream based on the real-time stream processing framework to generate a corresponding warning video or alarm information.

[0107] Specifically, real-time query processing: ① Message queue: In order to achieve real-time query processing, the system can introduce a message queue (such as Kafka) as data middleware. When a vehicle generates warning information, the background service sends the warning information to the Kafka queue. The video platform obtains the warning information in real time by subscribing to the messages in the Kafka queue and triggers the corresponding warning video generation process. The warning is used to feedback situations such as information backlog. The message queue Kafka is used to improve the real-time nature of information processing. This method can achieve low-latency transmission and efficient processing of warning information. ② Asynchronous processing: In order to improve concurrent processing capabilities, the video platform can use asynchronous processing to generate warning videos. When receiving warning information, the video platform puts the warning video generation task into the task queue and immediately returns a response to the caller. The background worker thread takes the task out of the task queue and executes the warning video generation process. This method can make full use of system resources and improve the throughput and response speed of the system. ③ Real-time stream processing: For scenarios that require real-time analysis of video data, a real-time stream processing framework (such as Apache Flink) can be introduced to process video data streams. The real-time stream processing framework Apache Flink supports high throughput, low latency, and precise state management, and can perform real-time analysis and processing of video data streams. By connecting the video data stream to Flink, abnormal situations in the video can be detected in real time, and corresponding warning videos or alarm information can be generated.

[0108] In this embodiment, data storage and query are key links in the warning video generation system for autonomous driving vehicles. By adopting a reasonable data storage solution, optimizing data query performance, and realizing real-time query processing, this embodiment can efficiently and stably run the multi-source data integration method for test drive scenarios and meet the needs of supervisors to view and analyze warning videos in real time.

[0109] It is understandable that if Figure 4 As shown, a specific test drive business scenario implementation case is taken as an example to illustrate the implementation process of the method of the present invention:

[0110] Step 1: Data collection: Capture test drive related data in real time from multiple HBase tables, such as customer information, vehicle information, test drive time, test drive mileage, etc.

[0111] Step 2: Data processing: Use Flink to clean, transform, and integrate data. For example, remove duplicate test drive records, convert test drive time into a unified format, and associate customer information with vehicle information.

[0112] Step 3: Data storage: Store the processed data in a Hive table and partition it by test drive time for subsequent query and analysis.

[0113] Step 4: Data application: Based on the data in the Hive table, provide intelligent decision support for the test drive business, such as evaluating the test drive effect, optimizing the test drive process, and predicting customer needs.

[0114] It should be noted that the above method of this embodiment has the following advantages: improving data processing efficiency: through Flink's real-time stream processing capability, the rapid integration of multi-source HBase data in the test drive scenario is realized, thereby improving data processing efficiency; ensuring data consistency: through data cleaning and standardized operations, data redundancy and errors are eliminated, and the consistency and accuracy of the integrated test drive data are ensured; enhancing business decision-making capabilities: based on the integrated test drive data, a more comprehensive and in-depth analysis perspective is provided for the test drive business, thereby enhancing the intelligence level of business decision-making; reducing operation and maintenance costs: by optimizing the reading, conversion and storage strategies of the test drive data, the complexity of the test drive data processing is reduced, and data processing resources are saved.

[0115] This embodiment provides a method for integrating multi-source data in a test drive scenario, including: capturing raw test drive data from multiple HBase tables in real time based on Flink technology; cleaning, converting and integrating the raw test drive data based on Flink's custom functions and conversion operations to obtain target test drive data; storing structured metadata and index data in the target test drive data in a preset unified database; storing the video file of the target test drive data in a distributed file system, and storing the preset key data of the target test drive data in a memory database; and performing test drive evaluation and optimization based on the stored target test drive data. In this embodiment, multi-source data of the test drive scenario is efficiently integrated based on Flink's big data processing technology. Through Flink's real-time stream processing capabilities combined with optimized data reading, conversion and storage strategies, fast, accurate and highly consistent integration of multi-source data is achieved, thereby providing strong support for decision-making in the test drive business.

[0116] In addition, an embodiment of the present invention further proposes a storage medium, on which a multi-source data integration program for test ride and test drive scenarios is stored. When the multi-source data integration program for test ride and test drive scenarios is executed by a processor, the steps of the multi-source data integration method for test ride and test drive scenarios described above are implemented.

[0117] Reference Figure 5 , Figure 5 This is a structural block diagram of an embodiment of a multi-source data integration system for test ride and test drive scenarios of the present invention.

[0118] like Figure 5 As shown, the test drive scenario multi-source data integration system includes:

[0119] A data capture module 10 is used to capture original test ride data from multiple HBase tables in real time based on Flink technology;

[0120] A data conversion module 20 is used to clean, convert and integrate the original test drive data based on Flink's custom functions and conversion operations to obtain target test drive data;

[0121] A first data storage module 30 is used to store structured metadata and index data in the target test drive data into a preset unified database;

[0122] A second data storage module 40 is used to store the video file of the target test drive data in a distributed file system, and store the preset key data of the target test drive data in a memory database;

[0123] The data analysis module 50 is used to perform test drive evaluation and optimization based on the stored target test drive data.

[0124] Specifically, Figure 3 As shown, the system architecture of the test drive scenario multi-source data integration system described in this embodiment may include four parts: data acquisition layer, data processing layer, data storage layer and data application layer. Among them, the data acquisition layer may include a data capture module 10, which is responsible for real-time capture of test drive related data from multiple HBase tables; the data processing layer may include a data conversion module 20, which uses Flink to clean, convert and integrate data; the data storage layer may include a first data storage module 30 and a second data storage module 40, which stores the processed data in a unified database for subsequent analysis; the data application layer may include a data analysis module 50, which provides intelligent decision support for the test drive business based on the integrated data (target test drive data).

[0125] It can be understood that the above method of this embodiment has the following advantages: improving data processing efficiency: through Flink's real-time stream processing capabilities, the rapid integration of multi-source HBase data in the test drive scenario is realized, thereby improving data processing efficiency; ensuring data consistency: through data cleaning and standardized operations, data redundancy and errors are eliminated, and the consistency and accuracy of the integrated test drive data are ensured; enhancing business decision-making capabilities: based on the integrated test drive data, a more comprehensive and in-depth analysis perspective is provided for the test drive business, thereby enhancing the intelligence level of business decision-making; reducing operation and maintenance costs: by optimizing the reading, conversion and storage strategies of the test drive data, the complexity of the test drive data processing is reduced, and data processing resources are saved.

[0126] In this embodiment, a multi-source data integration system for test drive scenarios is provided. Multi-source data of test drive scenarios are efficiently integrated based on Flink big data processing technology. Through Flink's real-time stream processing capabilities combined with optimized data reading, conversion and storage strategies, fast, accurate and highly consistent integration of multi-source data is achieved, thereby providing strong support for decision-making in test drive business.

[0127] It should be noted that, for technical details that are not described in detail in the embodiment of the multi-source data integration system for test drive scenarios, reference can be made to the multi-source data integration method for test drive scenarios described above provided in any embodiment of the present invention, and will not be repeated here.

[0128] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0129] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0130] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0131] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0133] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for integrating multi-source data in a test drive scenario, characterized in that: include: Capture raw test drive data from multiple HBase tables in real time based on Flink technology; Clean, transform and integrate the original test drive data based on Flink’s custom functions and transformation operations to obtain target test drive data; Storing structured metadata and index data in the target test drive data into a preset unified database; Storing the video file of the target test drive data in a distributed file system, and storing the preset key data of the target test drive data in a memory database; Test drive evaluation and optimization are performed based on the stored target test drive data.

2. The method according to claim 1, characterized in that The Flink technology is used to capture raw test drive data from multiple HBase tables in real time, including: Evenly divide the data into multiple Flink tasks according to the partitions of the HBase table; wherein each of the Flink tasks reads data from one or more partitions; Dynamically adjust the number and resource allocation of the Flink tasks through a load balancing algorithm to determine the partition that each Flink task corresponds to; Based on the Flink task, multiple HBase tables are read in parallel to obtain the original test drive data.

3. The method according to claim 1, characterized in that The Flink-based custom function and conversion operation cleans, converts and integrates the original test drive data to obtain the target test drive data, including: The original test drive data is filtered and corrected based on the custom function of Flink to obtain the cleaned test drive data; Performing data standardization processing on the cleaned test drive data based on the conversion operation of Flink to obtain standardized test drive data; The standardized test ride and test drive data are integrated to obtain target test ride and test drive data.

4. The method according to claim 3, characterized in that The Flink-based custom function filters and corrects the original test drive data to obtain cleaned test drive data, including: Identify and delete duplicate records of the original test drive data according to a unique index function in the database or write an SQL query statement to obtain cleaned data; Setting a threshold range, identifying the cleaned data based on the threshold range and correcting erroneous data to obtain corrected data; The corrected data is screened according to the query start and end time of the test drive and the frame number to retain the test drive analysis data and obtain the cleaned test drive data.

5. The method according to claim 3, characterized in that The Flink-based conversion operation performs data standardization processing on the cleaned test drive data to obtain standardized test drive data, including: Obtaining the timestamp of the cleaned test ride data; The timestamp is converted into a unified format based on a conversion operation of Flink to obtain first converted data; Obtaining speed and acceleration units of the first conversion data; The speed and acceleration units are standardized based on the conversion operation of Flink to obtain second conversion data; The data format of the second conversion data is standardized based on the conversion operation of Flink to obtain standardized test ride and test drive data.

6. The method according to claim 3, characterized in that The step of integrating the standardized test drive data to obtain target test drive data includes: Performing trajectory data integration on the standardized test ride and test drive data to obtain first integrated data; Acquire the ignition and shutdown status data of the first integrated data; The ignition on / off status data is associated and integrated with the timestamp, speed and acceleration data to obtain target test ride and test drive data.

7. The method according to any one of claims 1 to 6, characterized in that The test drive evaluation and optimization based on the stored target test drive data includes: Create indexes for key fields; When receiving a data query instruction, the statement of the data query instruction is automatically optimized based on the SQL query optimizer to obtain an optimized query statement; Querying the stored target test drive data according to the optimized query statement and the index, obtaining a query result and caching the query result; Test drive evaluation and optimization are performed based on the query results.

8. The method according to claim 1, characterized in that The method further comprises: Build a message queue as data middleware; When a test drive vehicle generates warning information, receiving the warning information based on the message queue; Obtain the warning information in real time by subscribing to the messages in the message queue; Based on the warning information, an asynchronous processing method is used to trigger a warning video generation process to obtain a video data stream; Build a real-time stream processing framework; The video data stream is analyzed and processed in real time based on the real-time stream processing framework to generate corresponding warning videos or alarm information.

9. A test drive scenario multi-source data integration system, characterized in that: include: Data capture module, which is used to capture raw test drive data from multiple HBase tables in real time based on Flink technology; A data conversion module is used to clean, convert and integrate the original test drive data based on Flink's custom functions and conversion operations to obtain target test drive data; A first data storage module is used to store structured metadata and index data in the target test drive data into a preset unified database; A second data storage module is used to store the video file of the target test drive data in a distributed file system, and store the preset key data of the target test drive data in a memory database; A data analysis module is used to perform test drive evaluation and optimization based on the stored target test drive data.

10. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a test drive scenario multi-source data integration program stored in the memory and executable on the processor, wherein the test drive scenario multi-source data integration program is configured to implement the test drive scenario multi-source data integration method according to any one of claims 1 to 8.

Images