Time-synchronized multi-source heterogeneous data playback method and storage medium

Abstract

The invention discloses a time-synchronized multi-source heterogeneous data playback method and a storage medium, and the method comprises the steps: dividing data into data of a database and a file, and establishing a corresponding data access interface; establishing a data model, and analyzing data of different structures according to the model; time synchronization of a multi-source data playback process is guaranteed by using a master-slave time axis mechanism and a condition waiting lock; and two data export interfaces, namely a network data export interface and a local API (Application Program Interface) are provided, so that data playback and data display are decoupled. The method solves the problems that a traditional data playback method is only suitable for a single data source or isomorphic data, data playback and analysis display are strongly coupled and the like, has adaptivity, can be suitable for playback of heterogeneous data from different data sources through two types of data access interfaces and establishment of different data models, and has good practicability.

Description

Technical Field

[0001] The present invention relates to the field of multi-source heterogeneous data management, and in particular to a time-synchronized multi-source heterogeneous data playback method and storage medium. Background Art

[0002] Data playback technology, as an important part of modern detection technology, is the follow-up and derivative of data acquisition technology. Combined with data acquisition technology and signal processing technology, it has been widely used in many fields such as radar, communication, hydroacoustic telemetry, remote sensing, and seismic survey. With the continuous improvement of people's demand for the performance of signal detection technology, how to quickly and accurately playback a large amount of historical data and provide effective historical data for signal processing technology has become one of the research focuses of signal detection technology.

[0003] In the actual test playback application scenario, due to the complexity of the test environment, it is necessary to simultaneously play back the trajectory data and various test information data during the test to be as close to the actual test situation as possible, so that the data to be played back after the test is data from different sources with different structures. The existing test data playback method only plays back data from a single data source, and does not consider the time synchronization of multi-source heterogeneous data playback. CN117406249A discloses an anti-UAV satellite navigation device and data playback method, which only supports navigation intermediate frequency data in a specific format; CN118069706A discloses a data recording and playback strategy implementation method, device and application, which only supports the recording and playback of data from a single source with time continuity. Therefore, it is meaningful to design a playback method that can handle the time synchronization of multi-source heterogeneous data.

[0004] In addition, the above data playback methods are generally strongly coupled with the data analysis and display tasks. For different playback data, using the same set of analysis and display methods cannot show the unique characteristics of different data. It is unrealistic to consider the display of all data. At the same time, too much display will make the display effect too messy. Therefore, data playback and data analysis and display need to be decoupled. Summary of the invention

[0005] The purpose of the present invention is to address the problems in existing experimental data playback technology, such as difficulty in handling time-synchronized playback of multi-source heterogeneous data and strong coupling with data analysis and display tasks, which lead to limited availability and versatility of data playback. The present invention provides a time-synchronized multi-source heterogeneous data playback method to improve the availability and versatility of data playback.

[0006] The technical solution of the present invention is as follows:

[0007] A time-synchronized multi-source heterogeneous data playback method fully considers the characteristics of multi-source heterogeneous data and the time synchronization mode to complete data playback, the method comprising:

[0008] Divide data into database and file data, and establish corresponding data access interfaces;

[0009] Establish a data model and parse data of different structures according to the model;

[0010] Use the master-slave timeline mechanism and conditional wait lock to ensure time synchronization of multi-source data playback process;

[0011] Provides two data export interfaces: network and local API, so that data playback and data display are decoupled.

[0012] By adopting the above technical solution, multi-source data is first accessed through the corresponding data interface, a data model of the corresponding data is established, and the original data is parsed through the data model to form multi-source heterogeneous playback data. The time synchronization mechanism of the master-slave time axis and the conditional wait lock and the data sending interface are used to realize the time synchronization playback of multi-source heterogeneous data.

[0013] Optionally, multi-source data includes:

[0014] Database data, which is generally stored in a relational database in the form of a table structure;

[0015] File data is generally stored in files in EXCEL, CSV, or XML data formats.

[0016] Optional access to multi-source data includes:

[0017] Access to database data: batch import of data can be completed by obtaining data through SQL query;

[0018] Access file data and complete batch import of data by traversing the files and extracting data according to the corresponding file format.

[0019] Optional parsing of heterogeneous data includes:

[0020] Establish corresponding data models according to data of different structures;

[0021] Map the fields in the original data format to the fields in the data required for playback.

[0022] Optionally, data playback controls include:

[0023] Create a main timeline;

[0024] Different data to be played back are controlled by different threads, and each thread establishes a time to identify the data from the time axis;

[0025] During the playback process, each thread will use the conditional wait lock to compare the data time with the main time. Only the data that reaches the corresponding time point can be played back, otherwise it will spin wait.

[0026] Optionally, data distribution includes:

[0027] Network protocol distribution can convert playback data into network message data distribution for remote analysis and display;

[0028] API interface distribution: distribute the playback data in the form of API to local analysis and display.

[0029] Optionally, the invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a time-synchronized multi-source heterogeneous data playback method described in the present invention.

[0030] The above solution of the present invention includes at least the following beneficial effects:

[0031] 1) By establishing two types of data interfaces and custom data models, the playback of multi-source heterogeneous data can be processed, solving the problem that traditional data playback methods are only applicable to single data sources or homogeneous data;

[0032] 2) By establishing a master-slave timeline and using conditional wait locks to control the parallel data playback process, the time synchronization of the parallel playback process of different data is ensured;

[0033] 3) Provides two data export interfaces: network and local API, so that playback data can be used for different data analysis and display on multiple terminals, solving the problem of strong coupling between data playback and analysis and display;

[0034] 4) The method is adaptive and can be applied to the playback of heterogeneous data from different data sources through two types of data access interfaces and the establishment of different data models. The invention has good practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is an overall flow chart of a time-synchronized multi-source heterogeneous data playback method of the present invention.

[0036] Figure 2 The present invention is a flowchart of a method for replaying multi-source heterogeneous data in a time-synchronous manner in an embodiment of the present invention.

[0037] Figure 3 It is a core flow chart of the playback process of a time-synchronized multi-source heterogeneous data playback method in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0039] See also Figure 2 and Figure 3 , a time-synchronized multi-source heterogeneous data playback method, the steps are as follows:

[0040] Step 1: Access multi-source data;

[0041] Step 2, parsing and preprocessing the data of different structures in the multi-source data to form data to be replayed;

[0042] Step 3, replay the data to be replayed, and control the time synchronization of the replay process.

[0043] Step 4: Distribute the data as needed. One is the network protocol, which can convert the playback data into network message data for remote analysis and display; the other is the API interface mode, which distributes the playback data in the form of API to local analysis and display.

[0044] A time-synchronized multi-source heterogeneous data playback method proposed in an embodiment of the present invention can simply and conveniently complete the time-synchronized playback of data from different sources and with different structures and achieve the decoupling of data analysis and display tasks through the access of multi-source data, the analysis of heterogeneous data, the control of the playback process, and the distribution of data.

[0045] In a possible implementation, the access of the multi-source data is of two types: one is the access of database data, which is generally real-time data collected by data acquisition software during the experiment and stored in a relational database in the form of a table structure; the other is file data, which is generally non-real-time data that can only be obtained after the experiment or experimental simulation data for comparison, and is stored in a file in EXCEL, CSV, or XML data format.

[0046] Specifically, to access database data, you first need to determine the database information and table information where the data is stored. Secondly, you need to clarify the names of the data columns that need to be obtained. Then, you need to splice SQL statements based on the table information and data column names. Finally, you need to connect to the database and execute SQL queries to obtain data to complete the batch import of data.

[0047] Specifically, to access file data, you first need to select the storage file, then you can perform corresponding data parsing according to its file format, then you need to clarify the arrangement format of the required extracted fields in the file, and finally traverse the file and extract data according to the arrangement format to complete the batch import of data.

[0048] In a possible implementation, the parsing of data of different structures requires the establishment of a corresponding data model for mapping fields in the original data format to fields in the data required for playback. At the same time, different data models can be established to process data with different key fields to ensure the integrity of key data.

[0049] Specifically, first customize the data model, establish the data model according to the playback needs, and establish the field mapping rules between the original data and the playback data; secondly, fill in the data model, parse the original data and fill in the data model according to the mapping rules, and obtain the playback data; finally, call the data model, store the playback data in the memory in chronological order for data playback calls.

[0050] In a possible implementation, the time-synchronized playback process control includes establishing a main timeline, and different data are controlled by different threads. Each thread has a time from the timeline to identify the data. During the playback process, each thread will use a conditional wait lock to compare the data time with the main time. Only data that reaches the corresponding time point can be played back. Its core workflow is as follows: Figure 2 shown.

[0051] Specifically, after the main thread selects the multi-source data to be played back and starts the playback process, it starts the main timeline and creates child threads for different data sources; each child thread starts from the timeline and calls the playback data in chronological order. When the main time point reaches the playback data time, the data is played back, otherwise the thread is spin-waited until the time point requirement is met.

[0052] In a possible implementation, the data distribution includes network protocol distribution and API interface mode distribution.

[0053] Specifically, the network protocol distribution first needs to specify the sending target IP and port and the message protocol (UDP / TCP), then assemble the playback data into messages of the corresponding protocol and serialize them into binary data, and finally actively distribute them.

[0054] Specifically, the API interface mode distribution first puts the playback data into a distributable queue, then provides a queue access interface, and finally the local analysis and display task that needs to use the data accesses the interface to obtain the data.

Claims

1. A time-synchronized multi-source heterogeneous data playback method, characterized in that: The steps include: S1, access to multi-source data; S2, analysis of heterogeneous data Establish corresponding data models for data of different structures, which are used to map fields in the original data format to fields in the data required for playback and process data with different key fields to ensure the integrity of key data; S3, data playback control Establish a main timeline, and use different threads to control different data. Each thread has a time to identify the data from the timeline. During the playback process, each thread will use the conditional wait lock to compare the data time with the main time. Only the data that reaches the corresponding time point can be played back, otherwise it will spin wait. S4, data distribution.

2. The time-synchronized multi-source heterogeneous data playback method according to claim 1, characterized in that: The multi-source data in step S1 includes database data and file data. The database data is real-time data collected by data acquisition software during the experiment and stored in a relational database in the form of a table structure; the file data is non-real-time data that can only be obtained after the experiment or experimental simulation data used for comparison.

3. The time-synchronized multi-source heterogeneous data playback method according to claim 2, characterized in that: The access to the database data first determines the database information and table information of the data storage, and then clarifies the data column name to be obtained, and then splices the SQL statement according to the table information and data column name, and finally connects to the database and executes SQL query to obtain data to complete the batch import of data.

4. The time-synchronized multi-source heterogeneous data playback method according to claim 2, characterized in that: The access of the file data first selects the storage file, then performs corresponding data analysis according to its file format, then clarifies the arrangement format of the required extracted fields in the file, and finally traverses the file and extracts data according to the arrangement format to complete the batch import of data.

5. The time-synchronized multi-source heterogeneous data playback method according to claim 2, characterized in that: The file data is stored in the file in EXCEL, CSV or XML data format.

6. The time-synchronized multi-source heterogeneous data playback method according to claim 1, characterized in that: In step S2, the establishment of the corresponding data model includes: (a) Establish field mapping rules between original data and playback data; (b) Data model filling: according to the mapping rules, the original data is parsed and the data model is filled to obtain the playback data; (c) Data model call, storing the playback data in time sequence to the memory for data playback call.

7. The time-synchronized multi-source heterogeneous data playback method according to any one of claims 1 to 6, characterized in that: In step S4, data distribution includes two modes: one is network protocol, which converts playback data into network message data for distribution and remote analysis and display; The other is the API interface mode, which distributes the playback data in the form of an API for local analysis and display.

8. The time-synchronized multi-source heterogeneous data playback method according to claim 7, characterized in that: Network protocol distribution includes: first, you need to specify the target IP and port as well as the message protocol UDP / TCP, then assemble the playback data into messages of the corresponding protocol and serialize them into binary data, and finally actively distribute them.

9. The time-synchronized multi-source heterogeneous data playback method according to claim 7, characterized in that: API interface mode distribution includes: first putting the playback data into the distributable queue, secondly providing the queue access interface, and finally obtaining the data through the local analysis and display task access interface that needs to use the data.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the time-synchronized multi-source heterogeneous data playback method according to any one of claims 1 to 9 are implemented.

Images