Vehicle-mounted system supporting multimodal real-time database extended playback and its implementation method

Abstract

This invention discloses an in-vehicle system and its implementation method that supports extended playback of multimodal real-time databases. The method first acquires multimodal real-time data through data acquisition, then preprocesses the multimodal data using distributed or multi-threaded processing to improve efficiency. Following this, it performs data error handling, monitors system performance indicators and resource utilization, and performs data transcoding and parsing. It also includes detection of transcoded data and extraction and correlation of auxiliary data information. Finally, it enables extended playback and visualization processing. This invention, when applied to an in-vehicle system, enables extended data playback and efficient processing to meet visualization requirements.

Description

Technical Field

[0001] This invention pertains to data processing technology, specifically relating to an in-vehicle system and its implementation method that supports multimodal real-time database extended playback. Background Art

[0002] Intelligent driving requires the collection of multimodal data (such as images, point clouds, sound, inertial navigation, etc.) of the vehicle's surrounding environment to achieve perception and decision-making. Due to the presence of multiple sensors, the unified processing, synchronization, and alignment of multimodal data affect the effectiveness of playback simulations and algorithm training.

[0003] Prior art methods similar to this invention include:

[0004] Single-modal data analysis tools: These tools are focused on analyzing and processing specific types of data, such as image analysis tools or bus signal analysis tools. They cannot process different types of data simultaneously.

[0005] Traditional database systems are primarily designed for processing structured data, such as tabular data. While they are very powerful at handling structured data, they are less capable of handling multimodal data. Traditional database systems may not be able to directly store and process unstructured data, such as images, audio, or video.

[0006] Data recorder playback technology: This focuses on recording and playing back vehicle driving data. These recorders typically record a wide range of information during vehicle operation, such as speed, acceleration, and braking. However, existing technologies face significant challenges in multimodal data playback and real-time scalability.

[0007] Existing technologies have the following drawbacks in large-scale real-time data playback:

[0008] Low processing efficiency: Multimodal data typically involves large volumes of data and complex data structures, such as images, audio, and video. Existing technologies are often inefficient in processing large-scale multimodal data and cannot meet real-time requirements.

[0009] Limited scalability: Traditional real-time databases have limited scalability. When the types and amounts of data increase, they cannot guarantee efficient data retrieval.

[0010] Traditional real-time databases are unsuitable for the requirements of intelligent driving in-the-loop simulation. They do not consider the data usage scenarios of intelligent driving, and their bottlenecks in data processing, storage, and analysis will directly affect the performance of intelligent driving in-the-loop simulation. Summary of the Invention

[0011] Purpose of the Invention: The present invention aims to solve the technical problems of multimodal real-time scalable database playback devices and systems. Accordingly, the first objective of the present invention is to provide an implementation method that supports multimodal real-time database extended playback. Based on the provided method, the second objective of the present invention is to provide an in-vehicle system that supports multimodal real-time database extended playback.

[0012] To achieve the above-mentioned objectives, the present invention adopts the following technical solution.

[0013] An implementation method supporting extended playback of multimodal real-time databases, comprising the following steps:

[0014] S1. Data Acquisition: Acquire multimodal data, including video, image, sound, and bus signal data, through the data bus;

[0015] S2. Preprocess the collected data using a thread pool: Organize and fragment the data into metadata, then divide it into several subsystems for distributed and multi-threaded concurrent processing, and simultaneously support horizontal and vertical scaling through the thread pool parameter configuration Config file.

[0016] S3. Data error handling: Perform integrity and correctness checks on the fragmented metadata. Metadata that fails the check is copied from the historical thread cache and the data fragmentation is retried until the number of failures exceeds the set K times. Then, a missing identifier is added to the dataset of the corresponding time period.

[0017] S4. Monitoring system performance indicators and resource utilization: If the number of missing identifiers exceeds Z within a period of time T, the reporting terminal will prompt that the corresponding data source or operating environment may be faulty.

[0018] S5. Data Fragment Index Location: Based on the index information added to each data fragment, including data timestamps, data source and environmental information, the target data fragment is located by searching the index using a tree structure.

[0019] S6. Parse and transcode the indexed data: Use data parsing algorithms to parse and transcode the collected raw data, transforming it into a data structure and format that is directly readable or easy to transmit;

[0020] S7. Transcoded Data Detection: Captures parsing errors and anomalies during data parsing and transcoding, and marks invalid or corrupted data segments;

[0021] S8. Data Auxiliary Information Extraction and Association: Extract metadata auxiliary information from the dataset, including timestamps, data sources, and environmental information, and bind the auxiliary information to the transcoded data. Based on the correlation of the data, extract relevant information and perform aggregated key-value association with the transcoded data.

[0022] S9. Playback Processing: Through multi-threaded parallel processing and optimized data storage structure, fast data playback and visualization processing of multi-source data are achieved.

[0023] Furthermore, the method includes the following two horizontal processing methods for the preprocessed metadata:

[0024] (a) Process the data according to the thread logic, send the metadata to the rendering module according to the occurrence time, and display it in a visual way after GPU rendering;

[0025] (b) Based on the data type, the original data in the metadata is fed back to the data bus or back to the model training according to the occurrence time, and the image recognition is achieved through deep learning algorithms.

[0026] Furthermore, the preprocessed metadata is compressed according to data type and stored in a local or cloud database.

[0027] Furthermore, the method also includes multimodal data visualization, specifically including:

[0028] Hierarchical visualization: Visualizing multimodal data according to different levels, with each level representing a different data modality;

[0029] Parallel coordinate graphs: These graphs use one or more parallel coordinate axes, each representing a data dimension. Users can select and adjust different coordinate axes to visualize modalities of interest.

[0030] Multidimensional visualization: mapping one or more data modalities to a multidimensional space, including parallel coordinate plots, radar charts or multidimensional scatter plots, or mapping several data modalities to a corresponding number of coordinate axes or dimensions;

[0031] Interactive visualization: This allows users to adjust the visualization perspective of data modalities, filter specific modal data, or change visualization parameters through interactive elements, including sliders, selectors, or stretch controls.

[0032] Based on the above method, an in-vehicle system supporting multimodal real-time database extended playback includes a data acquisition terminal, a data processing module, and a visualization terminal device. The data acquisition terminal is connected to an in-vehicle camera, millimeter-wave radar, lidar, and microphone via a wiring harness; then, the acquired data is transmitted to the data processing module through the vehicle bus interface.

[0033] The data processing module is used to execute the implementation method that supports multimodal real-time database extended playback;

[0034] The visualization terminal device includes an in-vehicle display screen, which is used to display data processed by the data processing module and supports playback.

[0035] Furthermore, the data processing module includes a wireless transmission module that offloads the data to be processed to a roadside cloud server for computation.

[0036] Furthermore, the data processing module includes an external mobile device to enable data playback and visualization.

[0037] Beneficial effects: Compared with the prior art, the significant effects of this invention include:

[0038] First, multimodal data visualization and playback: This invention can efficiently process the visualization and playback of multimodal data. It ensures the temporal consistency and accuracy of the data and supports scalable data playback for up to ten modalities.

[0039] Second, data integrity and anomaly handling mechanisms: During playback, this invention emphasizes ensuring the integrity of multimodal data. Through data verification and error handling mechanisms, this invention can ensure the integrity and traceability of data during playback.

[0040] Third, real-time performance and scalability: The playback system of this invention features real-time performance and scalability. By using distributed encoding and preprocessing methods in the vehicle-side database, repeated encoding and decoding during data transmission are reduced, saving data latency and overall computing power overhead, meeting the requirements of remote monitoring and real-time data service access, and providing fast and efficient multimodal data playback capabilities.

[0041] Fourth, cross-platform compatibility and interoperability: The playback device and system design of this invention features cross-platform compatibility and interoperability. It can be easily integrated with different hardware and software environments. It can also be implemented on independent hardware without consuming existing hardware resources. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the data flow in the method described in this invention;

[0043] Figure 2 It is a metadata structure diagram;

[0044] Figure 3 This is a schematic diagram of the framework of the system constructed by the present invention. Detailed Implementation

[0045] To illustrate the technical solutions disclosed in this invention in detail, further descriptions are provided below in conjunction with the accompanying drawings.

[0046] This invention aims to solve the technical problems of implementing multimodal real-time scalable database playback and its application in vehicles. Specifically, the key technical problems solved by this invention are as follows.

[0047] Multimodal data processing: Traditional database systems primarily focus on the storage and retrieval of text data, while this invention aims to process data in multiple modalities, including text, images, audio, and video. Therefore, this invention addresses the challenge of efficiently processing data of different modalities to ensure that the data remains intact and accurate during real-time playback.

[0048] Real-time data transmission: Real-time playback requires rapid data transmission to ensure that the playback device can instantly acquire and display data from the database. Therefore, this invention addresses the problem of real-time data transmission, including how to achieve fast and stable data transmission through a high-performance data transmission channel.

[0049] Reduce data transcoding at both ends of the vehicle and cloud: Support multimodal data preprocessing on the vehicle side to avoid repeated encoding and decoding during transmission and processing, and support large-scale multimodal data playback requirements.

[0050] Scalability and parallel processing: In order to meet the ever-increasing data demand, this invention needs to solve how to design a scalable system architecture that enables multiple playback devices and database servers to work in parallel and provides scalable and high-performance playback capabilities.

[0051] Software-in-the-loop (SIL) testing data backfeeding: Multimodal data is replayed sequentially according to the actual occurrence time, serving as data input for software testing. This can be used for model training and simulation testing. Model training refers to the deep learning model in the vehicle-side autonomous driving module, which uses image recognition such as cars, pedestrians, and lane lines to provide the output results to subsequent algorithm modules for data processing in autonomous driving.

[0052] A statistical analysis module is introduced: This module enables statistical analysis of multimodal data, extracting key indicators and features for a deeper understanding and analysis of the data. Through this module, users can obtain various statistical information about multimodal datasets, such as frequency distribution, mean, and variance. This statistical information helps users understand the distribution, trends, and changes in the data. Deep learning algorithms and models are used for more advanced analysis and mining of multimodal data.

[0053] Combination Figure 1 and Figure 2 As shown, the present invention provides an implementation method for supporting multimodal real-time database extended playback, which includes the following steps:

[0054] Step 1: Connect the wiring harness: power supply and data terminal sensor devices, including cameras, millimeter-wave radar, lidar, microphones, CAN bus signals, etc.

[0055] Deploy thread pools: used for unified organization and management of multimodal data, as well as unified processing and replay of preprocessed metadata.

[0056] Data acquisition: Acquire multimodal data from the abstract data bus, including video, images, sound, bus signals, and other data.

[0057] Step 2: Preprocess the collected data using a thread pool, organizing and fragmenting data from different sources into metadata. In addition to the data content, the metadata should also include information such as data type, time, identifier, data length, checksum, and version.

[0058] Data preprocessing is divided into multiple subsystems. Distributed processing and multi-threaded concurrency technologies are employed to improve the speed of data preprocessing, querying, rendering, and playback. Furthermore, horizontal and vertical scaling is supported through thread pool parameter configuration files.

[0059] Step 3: Data error handling. Perform integrity and correctness checks on the fragmented metadata. Metadata that fails the check is copied from the historical thread cache and the data fragmentation is retried until the number of failures exceeds the set K times. Then, add a missing identifier to the dataset of the corresponding time period.

[0060] Step 4: Monitor system performance indicators and resource utilization. If the number of missing identifiers exceeds Z within a period of time T, the reporting terminal will prompt that the corresponding data source or operating environment may be malfunctioning.

[0061] Step 5: Based on the index information added to each data segment, including data timestamps, data sources, environmental information, etc., use a tree structure search index to locate the target data segment.

[0062] Step 6: Data transcoding and parsing: The indexed data is parsed and transcoded. Data parsing algorithms are used to parse and transcode the collected raw data, transforming it into a data structure and format that is directly readable or easy to transmit.

[0063] Step 7, Transcoded Data Detection: Capture parsing errors and anomalies during data parsing and transcoding, and mark invalid or corrupted data segments.

[0064] Step 8: Extract and Associate Data Auxiliary Information: Extract metadata auxiliary information from the dataset, including timestamps, data sources, and environmental information. Bind this additional information to the transcoded data. Based on data correlation, extract relevant information and perform aggregated key-value association with the transcoded data.

[0065] Step 9: Fast playback and multi-source data visualization and statistical analysis: Through multi-threaded parallel processing and optimized data storage structure, fast playback of data and visualization of multi-source data can be achieved.

[0066] Fast replay: Replay data is distributed across multiple concurrently executed subtasks within a thread pool to process data from multiple data sources simultaneously. During replay, data caching and pre-fetching mechanisms are used to pre-load the data to be replayed into memory, avoiding frequent disk read operations and thus reducing I / O latency. For large-scale datasets, data decoding can be performed during replay to reduce storage space and accelerate replay speed.

[0067] Multi-source data visualization and statistical analysis: This involves aggregating and correlating data from multiple data sources, extracting data correlation features and metadata, and then using visualization tools and techniques to present the data in chart form. For example, video data can be rendered using GPUs, signal data can be displayed using line charts, and statistical correlation analysis can be presented and analyzed using bar charts.

[0068] The method proposed in this invention supports querying and playback of multimodal data sources, including processing of various data sources such as video, images, audio, and bus signals. By cascading metadata processing of different data sources, interaction and compatibility between different data sources are achieved. Unified organization and management of different data sources can improve the efficiency and reliability of data querying and playback.

[0069] The data processing involves the following steps:

[0070] Metadata Information: Examine the metadata information of the input data. This information may include file headers, file structure, or specific identifiers. Based on the metadata information, file format, or specific identifiers of the input data, the type of data source can be determined, such as video, image, sound, or bus signal.

[0071] Data source classification: Data sources are classified according to their type so that specific processing and management can be carried out for different types of data sources in the future.

[0072] Internal structure analysis: For certain data formats, their format type can be determined by analyzing their internal structure. For example, the format of video data can be determined by parsing the file's frame structure, encoding method, and frame rate.

[0073] Compare with target system requirements: Compare the identified data format with the compatibility requirements of the target system to verify the compatibility standards.

[0074] Version check: If the target system requires a specific version of the data format, it is necessary to check whether the identified data format version is consistent with the version required by the target system.

[0075] The optimization design regarding scalability also includes the following processes:

[0076] As data volume increases, the technical solution provided by this invention supports dividing the dataset into multiple subsystems for processing. In processing multimodal data sources, it supports format expansion and conversion, employs distributed processing and multi-threaded concurrency technologies, and enables fast querying and playback of datasets, improving the system's scalability and efficiency. Simultaneously, the system also supports horizontal and vertical scaling, configurable according to the user's actual needs and scale.

[0077] (1) In dataset partitioning, the dataset is divided into multiple logical partitions or subsystems according to its size and nature. Then, a suitable data partitioning strategy is selected, such as partitioning by time, geographical location, data type, etc.

[0078] (2) In the design of distributed processing technology and architecture, as well as concurrency optimization, the first step is to design the distributed processing architecture, including data partitioning, task allocation, and data communication. Then, based on system requirements and the characteristics of data processing tasks, a suitable multi-threaded concurrency strategy is designed. Subsequently, mechanisms such as thread pools and task queues are considered to ensure the efficient concurrent execution of data processing tasks.

[0079] (3) Error handling and fault tolerance mechanisms: First, design error capture and exception handling mechanisms to ensure the stability and reliability of the system. Then, add fault tolerance mechanisms such as data loss, failure retry, and data recovery to prevent system failures or interruptions.

[0080] (4) Monitoring and Optimization: Design mechanisms for system monitoring and performance tuning, such as logging, metric monitoring, and automated adjustments. Then analyze system performance metrics and optimize system configuration and resource utilization in a timely manner to maintain system efficiency and scalability.

[0081] (5) Data Indexing and Location: This module enables indexing and location of data collected in the intelligent driving system. During data collection, index information is added to each data segment, including data timestamps, vehicle status, vehicle information, and environmental information, to facilitate rapid location and retrieval later. Simultaneously, this module can also manage large datasets in blocks to improve retrieval efficiency. Specific steps are as follows.

[0082] (5.1) Data Index Design

[0083] Index field definition: Determine the fields that need to be indexed. These fields should include key attributes such as data timestamp, vehicle status, vehicle information, and environmental information.

[0084] Index structure selection: Taking into account both query efficiency and index storage costs, select an appropriate index structure to support fast data retrieval and location.

[0085] (5.2) Data positioning implementation

[0086] Data slicing: Managing large datasets by dividing them into logical blocks. This improves fine-grained access and location of data.

[0087] Data block indexing: Add index information to each data block, including the start and end timestamps of the data block, as well as other key attributes. This allows for quick location and retrieval of data blocks.

[0088] Data search algorithm: Design a suitable search algorithm to quickly find data blocks that meet the query conditions and index information.

[0089] (5.3) Data retrieval and analysis:

[0090] Query Interface Design: Design a user-friendly query interface that allows users to retrieve and locate data for a specific time period or a specific type by specifying query conditions.

[0091] Efficient data retrieval: Utilizing index structures and data block indexes, efficient data retrieval and location are achieved to support rapid analysis and processing of large datasets.

[0092] Data analysis tools: Provide data analysis tools and algorithms for analyzing and mining retrieved data to extract useful information and patterns.

[0093] (6) Data transcoding and parsing: This module parses and transcodes the indexed data. It uses data parsing algorithms to parse and transcode the collected raw data, transforming it into a readable data structure and format. The specific steps are as follows:

[0094] (6.1) Data Analysis Design

[0095] Data format recognition: By combining the attributes, format, or protocol of the data, determine the appropriate parsing method and algorithm.

[0096] Data segmentation: For large datasets or continuous data streams, they are segmented for parsing and transcoding segment by segment. This improves parsing efficiency and stability.

[0097] (6.2) Data transcoding implementation

[0098] Format conversion: Convert raw data into a specific data format supported by the system, according to the system's needs. For example, convert text data to JSON format, and image data to JPEG or PNG format.

[0099] Data structuring: Transforming raw data into a readable data structure, such as object-oriented data structures, tables, or relational database schemas. This facilitates subsequent data analysis and processing.

[0100] (7) Data quality inspection

[0101] Error handling: An error handling mechanism should be incorporated into the data parsing and transcoding process to promptly capture and handle parsing errors and abnormal situations. For example, invalid or corrupted data segments should be marked or discarded.

[0102] Data validation: Validating the transcoded and parsed data ensures its integrity and accuracy. Data validation rules and algorithms can be used to check whether the data meets the expected format and content.

[0103] (8) Data-assisted information extraction:

[0104] Metadata extraction: Extracting metadata from the raw data, including timestamps, vehicle status, vehicle information, environmental information, etc. This metadata can be bound to the transcoded data as additional information to enrich the data content and context.

[0105] Related information extraction: Based on the correlation between data, relevant information is extracted and associated with the transcoded data. For example, image data can be associated with sensor data to achieve more comprehensive data analysis and processing.

[0106] (9) Fast playback and multi-source data visualization and statistical analysis: Through multi-threaded parallel processing and optimized data storage structure, fast playback of data and visualization of multi-source data can be achieved, while significantly improving the efficiency and flexibility of data processing.

[0107] (9.1) Fast playback function

[0108] Multi-threaded parallel processing: By using multi-threading technology, the data playback operation is divided into multiple concurrently executed subtasks to speed up data playback. This allows data from multiple data sources to be processed simultaneously, improving playback efficiency.

[0109] Data caching and prefetching: During playback, data caching technology and prefetching mechanism are used to load the data to be played back into memory in advance, avoiding frequent disk read operations, thereby reducing I / O latency and speeding up data playback.

[0110] Compression and decompression: For large-scale datasets, data compression techniques can be used to compress and store the data, thereby reducing storage space and speeding up data playback.

[0111] (9.2) Visualization and statistical analysis of multi-source data

[0112] Data aggregation and correlation: Aggregating and correlating data from multiple data sources, combining and matching corresponding data to obtain a more complete and consistent data view.

[0113] Visualization charts and graphical interfaces: Utilize visualization tools and technologies to present data in chart form, enabling users to understand and analyze data more intuitively. Examples include using scatter plots, bar charts, and line charts to display data.

[0114] Data statistics and analysis algorithms: These algorithms are used to statistically analyze the replayed data and extract useful information and patterns. For example, they can calculate parameters such as recognition rate, average speed, travel distance, and energy consumption to detect abnormal behaviors and patterns.

[0115] (9.3) Optimize data storage structure

[0116] Data indexing and compression: Use appropriate data indexing techniques to accelerate data retrieval and access. Meanwhile, for large datasets, consider using data compression techniques to reduce storage space.

[0117] Data sharding and distributed storage: Divide data into multiple shards and distribute the shards across multiple storage nodes to improve the parallelism and performance of data read and write operations.

[0118] Data partitioning and data redundancy: Based on the characteristics and access patterns of the data, data partitions are rationally divided, and data redundancy mechanisms are adopted to improve the availability and reliability of the data.

[0119] Combination Figure 3 As shown, an in-vehicle system supporting multimodal real-time database extended playback includes a data acquisition terminal, a data processing module, and a visualization terminal device. The data acquisition terminal is connected to an in-vehicle camera, millimeter-wave radar, lidar, and microphone via wiring harnesses; then, the acquired data is transmitted to the data processing module through a vehicle bus interface; the data processing module executes the implementation method supporting multimodal real-time database extended playback; the visualization terminal device includes an in-vehicle display screen for displaying the data processed by the data processing module and supports playback operations.

[0120] Furthermore, the protocol for the method and system is designed for scalability and compatibility as follows.

[0121] Version control: Compatibility is achieved by introducing version numbers into the protocol. When the protocol is upgraded or new features are added, new fields or message types can be introduced in the new version, and a version number can be specified in the protocol for identification, ensuring that different versions of the system are compatible with each other. This allows older versions of the system to continue using the old protocol, while newer versions of the system use the new protocol, thus achieving scalable compatibility of the protocol.

[0122] Extended fields: Compatibility is achieved by defining extended fields in the protocol. When new features or data items need to be added, dedicated extended fields are reserved in the protocol for future population. This allows older systems to ignore unknown extended fields, while newer systems can parse and use them, achieving scalable compatibility for the protocol.

[0123] Optional Fields: Compatibility is achieved using optional fields. By making certain fields or data items optional, older systems can selectively ignore these fields, while newer systems can use them. This allows new functionality or data items to be added to the protocol without breaking existing systems.

[0124] Adapters: Adapters are used to connect systems with different versions or protocols, achieving compatibility. As a separate layer, the adapter converts incompatible protocols into compatible ones. By performing protocol conversion at the adapter layer, different systems can use different protocols but still communicate and interact with each other.

Claims

1. A method for supporting extended playback of a multimodal real-time database, characterized in that, The method includes the following steps: S1. Data Acquisition: Acquire multimodal data, including video, image, sound, and bus signal data, through the data bus; S2. Preprocess the collected data using a thread pool: Organize and fragment the data into metadata, then divide it into several subsystems for distributed and multi-threaded concurrent processing, and simultaneously support horizontal and vertical scaling through the thread pool parameter configuration Config file. S3. Data error handling: Perform integrity and correctness checks on the fragmented metadata. Metadata that fails the check is copied from the historical thread cache and the data fragmentation is retried until the number of failures exceeds the set K times. Then, a missing identifier is added to the dataset of the corresponding time period. S4. Monitoring system performance indicators and resource utilization: If the number of missing identifiers exceeds Z within a period of time T, the reporting terminal will prompt that the corresponding data source or operating environment may be faulty. S5. Data Fragment Index Location: Based on the index information added to each data fragment, including data timestamps, data source and environmental information, the target data fragment is located by searching the index using a tree structure. S6. Parse and transcode the indexed data: Use data parsing algorithms to parse and transcode the collected raw data, transforming it into a data structure and format that is directly readable or easy to transmit; S7. Transcoded Data Detection: Captures parsing errors and anomalies during data parsing and transcoding, and marks invalid or corrupted data segments; S8. Data Auxiliary Information Extraction and Association: Extract metadata auxiliary information from the dataset, including timestamps, data sources, and environmental information, and bind the auxiliary information to the transcoded data. Based on the correlation of the data, extract relevant information and perform aggregated key-value association with the transcoded data. S9. Playback Processing: Through multi-threaded parallel processing and optimized data storage structure, fast data playback and visualization processing of multi-source data are achieved.

2. The implementation method for supporting multimodal real-time database extended playback according to claim 1, characterized in that, The preprocessed metadata includes the following two horizontal processing methods: (a) Process metadata according to data type and thread logic, send the metadata to the rendering module according to the occurrence time, and display it in a visual way after GPU rendering; (b) Based on the data type, the original data in the metadata is fed back to the data bus or back to the model training according to the occurrence time, and the image recognition is achieved through deep learning algorithms.

3. The implementation method for supporting multimodal real-time database extended playback according to claim 1, characterized in that, The preprocessed metadata is compressed according to data type and stored in a local or cloud database.

4. The implementation method for supporting multimodal real-time database extended playback according to claim 1, characterized in that, This method also includes multimodal data visualization, specifically including: Hierarchical visualization: Visualizing multimodal data according to different levels, with each level representing a different data modality; Parallel coordinate graphs: These graphs use one or more parallel coordinate axes, each representing a data dimension. Users can select and adjust different coordinate axes to visualize modalities of interest. Multidimensional visualization: mapping one or more data modalities to a multidimensional space, including parallel coordinate plots, radar charts or multidimensional scatter plots, or mapping several data modalities to a corresponding number of coordinate axes or dimensions; Interactive visualization: This allows users to adjust the visualization perspective of data modalities, filter specific modal data, or change visualization parameters through interactive elements, including sliders, selectors, or stretch controls.

Images