A Time Synchronization Method for Multi-Stream Data in Flight Tests

Abstract

The present invention relates to a flight test multi-stream data time synchronization method, comprising: S1: uniformly managing the flight test multi-stream data through a flight test multi-stream data time synchronization tool; S2: the synchronization tool uses a time series database to store the flight test multi-stream data and provides a time series retrieval capability; S3: the airborne audio and video data stream is stored in a file format in a data frame sequence; S4: the synchronization tool uses a CNN-like algorithm to identify the image timestamp of the airborne video data stream on a trained model to obtain a video image time series; S5: integrating the time series of the flight test multi-stream data, setting the absolute time and the reference time stream, and the remaining multi-streams are time synchronized according to the reference time stream; S6: replaying the flight test process. The present invention provides a high-precision time alignment and delay method for the time synchronization of the flight test multi-stream data, and completes the playback and display of the flight test data through the multi-stream data time synchronization tool.

Description

Technical Field

[0001] The present invention relates to the technical field of fault prediction and health management (PHM) in flight tests and equipment maintenance and support, and in particular to a flight test multi-stream data time synchronization method. Background Art

[0002] The flight test data collected during the flight test is a typical streaming data based on time sequence, such as flight parameter data, bus data, audio and video data, PCM data, etc., which are data that record corresponding values ​​in a sequence of time stamps. Since each type of flight test data comes from a different collector, and the collection frequency of each parameter is different from the time source used by the collector, it is often the case that the multi-stream data is not synchronized in time.

[0003] When flight test engineers and designers perform professional analysis on flight test data, they often need to use data streams from different collectors. If the parameters between different streams are not aligned in time, it will affect the analysis results and reduce the analysis efficiency.

[0004] When replaying the flight process, the flight parameter data, airborne audio and video data, and telemetry data recorded by the flight recorder are needed to achieve the purpose of truly restoring the flight process. These test flight data involved in the playback must be time-aligned.

[0005] The flight parameter data and telemetry data collected in flight tests are usually scattered in multiple channels to record thousands of parameter data. The different data streams and parameters of different channels are connected to different timing signals, and the parameter sampling rates and parameter word distributions are different, which will cause time inconsistency. They need to be output after time synchronization to ensure that the parameter values ​​in the same stream are time-aligned.

[0006] The absolute time of flight is usually not recorded in the onboard video data frames of flight tests. The absolute time is loaded and displayed in the image frame as a watermark. It is necessary to obtain the absolute time information in the image through machine vision algorithms to overcome the difficulty in obtaining flight time information from video images in flight tests.

[0007] The onboard multi-stream time alignment method for flight tests requires simultaneous time comparison and alignment of the timestamps recorded by the flight parameter data, telemetry data, and onboard audio and video data. It needs to rely on the time and time alignment algorithm of the reference data stream to quickly and accurately align the data.

[0008] The time-aligned data is displayed in a visual display control, allowing the flight test engineer to quickly and realistically restore the flight process, achieving the purpose of accurately completing the flight test data analysis. At present, there is no time alignment software suitable for flight test engineers, and there is an urgent need to develop and break through related technologies. Summary of the invention

[0009] In order to solve the above technical problems, the present invention provides a flight test multi-stream data time synchronization method.

[0010] The technical solution of the present invention is: a flight test multi-stream data time synchronization method, the method comprising the following steps:

[0011] Step S1: The flight test multi-stream data includes a flight test telemetry data stream, an airborne flight parameter data stream, an airborne audio data stream, and an airborne video data stream, and the flight test multi-stream data is uniformly managed by a flight test multi-stream data time synchronization tool;

[0012] Step S2: The flight test multi-stream data time synchronization tool uses a time series database to store the flight test telemetry data stream and the airborne flight parameter data stream. The time series database provides time series retrieval capabilities.

[0013] Step S3: The airborne audio data stream and the airborne video data stream are stored in a file format in a data frame sequence;

[0014] Step S4: The flight test multi-stream data time synchronization tool uses a CNN-like algorithm to identify the image timestamps of the airborne video data stream on the trained model to obtain a high-precision video image time series;

[0015] Step S5: Integrate the time series of multi-stream data of the flight test, set the absolute time and the reference time stream, and synchronize the remaining multi-stream data according to the reference time stream;

[0016] Step S6: After the time synchronization is completed, the flight test process is replayed in the flight test playback display screen in the flight test multi-stream data time synchronization tool.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] 1. The present invention discloses a time synchronization method for multi-stream data of a flight test, which can perform time synchronization alignment on parameter data of flight parameter data and telemetry data under multiple channels, and supports time alignment of multiple types of data such as flight parameter data, telemetry data, and airborne audio and video data, with high alignment accuracy.

[0019] 2. The present invention discloses an absolute time extraction method for airborne video data, which solves the problem that the absolute time of flight is not recorded in the data frame and cannot be synchronized with other data streams. The flight test multi-stream data time synchronization tool loads standard and proofread image samples to train a CNN-like algorithm, allowing it to adapt to the influence of the airborne cockpit environment, aircraft system instrument environment, and extravehicular environment on the timestamp in the image. It can accurately identify the timestamp information. When the absolute time information is missing from the video frame, the absolute time information of the entire video is supplemented, the absolute start and end time of the frame is recorded in the metadata, and the full sequence timestamp and the image frame number corresponding to the timestamp are also recorded in the time series database. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a method for time synchronization of multi-stream data of a flight test according to an embodiment of the present invention;

[0021] Figure 2 The present invention is a structural block diagram of a method for time synchronization of multi-stream data in a flight test according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention provides a flight test multi-stream data time synchronization method, which provides efficient and accurate application services for flight test review and playback of large and complex equipment and PHM fault analysis business.

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following is a specific implementation, combined with the attached Figure 1-2 , the present invention is further described in detail.

[0024] The present invention provides a flight test multi-stream data time synchronization method, the process of the method is as follows Figure 1 As shown, the method is composed as Figure 2 As shown, the following steps are included:

[0025] Step S1: The flight test multi-stream data includes the flight test telemetry data stream, the airborne flight parameter data stream, the airborne audio data stream and the airborne video data stream, which are uniformly managed by the flight test multi-stream data time synchronization tool;

[0026] Step S2: The flight test multi-stream data time synchronization tool uses a time series database to store the flight test telemetry data stream and the airborne flight parameter data stream. The time series database provides efficient time series retrieval capabilities.

[0027] Step S3: The airborne audio data stream and the airborne video data stream are stored in a file format in a data frame sequence;

[0028] Step S4: The flight test multi-stream data time synchronization tool uses a CNN-like algorithm to identify the image timestamps of the airborne video data stream on the trained model to obtain a high-precision video image time series. The specific steps are as follows:

[0029] Step S41: pre-processing the airborne video sample image data set;

[0030] Use the annotation tool to annotate the timestamp area in the sample image in the form of text box coordinates. The annotation method is:

[0031] (1) Extract key image frames from the video;

[0032] (2) defining a display area of ​​a timestamp in a key image frame;

[0033] (3) Label the text information corresponding to the timestamp. The labeled samples should include the normally displayed timestamp and the distorted timestamp to generate a data set;

[0034] Step S42: training a video timestamp recognition model;

[0035] (1) The CNN-like algorithm is composed of CNN+RNN, where CNN uses the VGG-16 model. It extracts local features from airborne video images through convolution kernels, performs downsampling (i.e., reduces the size of the image to extract key features), and preprocesses the input image, including adjusting the image size to ensure that the length and width are multiples of 32. This can reduce the pixel misalignment problem that may occur during the segmentation process, and reduces the dimension of the feature map through the pooling layer. These downsampled feature maps are then upsampled (enlarged) and combined with the original feature maps. After this processing, the algorithm generates two feature maps: Region Score Map (region score map: shows the probability of the center area of ​​each character in the image and scores the position of each character in the image) and Affinity Score Map (affinity score map: represents the probability of the center of the adjacent character area, used to evaluate the connection strength between characters and help the algorithm determine which characters are connected to each other). Together, these two score maps help the algorithm accurately identify and locate text in images in the case of complex layouts or irregular text arrangements;

[0036] (2) The RNN algorithm mainly recognizes curved text. By associating the text order, it helps the machine understand the text context, predicts the feature sequence, learns each feature vector in the sequence, and outputs the predicted label (true value) distribution.

[0037] Step S43: training a CNN-like model;

[0038] Arrange the sample images in the correct time sequence, configure the relevant parameters of the training CNN algorithm, and start model training;

[0039] Step S44: Load the model to extract the timestamp;

[0040] Place the obtained CNN-like algorithm model into the project; each time timestamp recognition is performed, the user first defines the timestamp area position, the tool will perform image extraction and region segmentation on the video data, perform prediction and recognition on the segmented image, and obtain the timestamp of the data frame;

[0041] Step S45: establish a corresponding relationship between the data frame and the timestamp, and store them uniformly in a time series database;

[0042] Step S5: Integrate the time series of the multi-stream data of the flight test, set the absolute time and the reference time stream, and synchronize the remaining multi-streams according to the reference time stream; the time synchronization algorithm is as follows:

[0043] Step S51: supporting alignment of timestamps in different time formats, including character formats such as "yy-MM-dd-HH-mm-ss-sss" and millisecond integer numbers "13947712990", with optional formats;

[0044] Step S52: Setting a reference time stream, comparing the rest of the data streams according to the start and end time, sampling frequency, and timestamp of the reference time stream, and making a secondary adjustment of the absolute time of the reference time;

[0045] Step S53: setting the start and end time and sampling rate of the remaining aligned time streams;

[0046] Step S54: Align the flight test multi-stream data according to the time alignment algorithm. The time alignment algorithm is as follows:

[0047] Step S541: Based on the reference time stream, compare the start and end times of the remaining multiple streams to find the start time position and end time position of the remaining data streams to start the comparison;

[0048] Step S542: Based on the reference time stream, compare the sampling rates of the remaining multiple streams, calculate the sampling rate difference, and obtain the number of points that need to be interpolated;

[0049] R d =R t / R s -1

[0050] In the above formula, R d is the sampling rate difference, R t For high sampling rate, R sFor low sampling rate, after calculating the sampling rate difference, if it is greater than the benchmark time stream sampling rate, then do the jump point calculation, the number of jump points is R d ; If it is less than the base time stream sampling rate, then interpolation is performed, and the number of interpolation points is R d , interpolation uses linear / non-linear interpolation.

[0051] Step S543: Obtain the timestamp of the reference time stream and compare it with the timestamps of the remaining data streams. Since the flight test data is accurate to milliseconds, it is necessary to compare the time from milliseconds. The alignment method is as follows:

[0052] (1) The sampling rates of all data streams have reached the same level;

[0053] (2) Take the starting timestamp of the reference time stream, find the alignment timestamps of all other streams according to the minimum difference method, and replace their timestamps with the timestamp of the reference time stream;

[0054] (3) After completion, take the next timestamp of the reference time stream and follow the method (2) until the end;

[0055] (4) finally obtaining aligned multi-stream data;

[0056] Step S6: After the time synchronization is completed, the flight test process is replayed in the flight test playback display screen in the flight test multi-stream data time synchronization tool.

[0057] The step S1: the flight test multi-stream data includes the flight test telemetry data stream, the airborne flight parameter data stream, the airborne audio data stream and the airborne video data stream, which are uniformly managed by the flight test multi-stream data time synchronization tool, specifically including:

[0058] It provides the import, storage, processing and query functions of flight test telemetry PCM data stream and iNet data stream. Among them, the telemetry PCM data stream and iNet data stream are processed according to the IRIG-106 standard, the synchronization word, full frame and subframe of the fixed frame structure, and support the extraction of processing parameters by time period / full process. The time provides range selection according to the time extracted from the PCM frame, and supports the processing of the whole time period. At the same time, it provides parameter selection function, and users can select / select all parameters for output;

[0059] It provides the functions of importing, storing, processing and querying flight parameter data streams. By decoding the flight parameter data stream, the timestamp in the data frame is obtained, and the timestamp sequence of the flight parameter data parameters is established;

[0060] Provides the functions of importing, storing, processing and querying airborne audio. Supports airborne audio formats including MP3, WMA, WAV, as well as encrypted audio data processing and parsing, and extracts the channel and track time length information in the audio;

[0061] Provides the functions of importing, storing, processing and querying airborne videos. Supports airborne video encoding formats including H.264 and H.265. During the processing, the DTS (Decoding Time Stamp) and PTS (Presentation Time Stamp) information can be extracted when decoding the video, and used as the basic start and end time information of the video data;

[0062] Step S2: The flight test multi-stream data time synchronization tool uses a time series database to store the flight test telemetry data stream and the airborne flight parameter data stream. The time series database provides efficient time series retrieval capabilities, specifically including:

[0063] A time series database is used to store the timestamps and parameter values ​​of the flight test telemetry data stream and the airborne flight parameter data stream, a time series two-dimensional table structure is established according to the timestamps and parameter values, and a data index is established on the timestamp;

[0064] Provides engineering value data query interface for flight test telemetry data stream and airborne flight parameter data stream, supporting conditional retrieval by parameters and all time periods;

[0065] In the flight test multi-stream data time synchronization tool, the airborne audio data stream and the airborne video data stream are stored in file format in the order of data frames. The airborne audio data stream is stored in the file + metadata mode. The file system stores audio and video data. The metadata provides information extension for the audio and video data, recording the audio and video name, size, format, length, frame start and end time information;

[0066] The absolute time of flight is usually not recorded in the airborne video data frame. The absolute time is loaded and displayed in the image frame in the form of a watermark. According to the flight test multi-stream data time synchronization tool, the flight test multi-stream data time synchronization tool trains the CNN algorithm by loading standard and proofread image samples to adapt it to the impact of the airborne cockpit environment, aircraft system instrument environment, and extravehicular environment on the timestamp in the image. It can accurately identify the timestamp information. In the case that the video frame lacks the absolute time information, the absolute time information of the entire video is supplemented, the absolute start and end time of the frame is recorded in the metadata, and the full sequence timestamp and the image frame number corresponding to the timestamp are recorded in the time series database.

[0067] Perform time synchronization and alignment on the multi-stream data of the flight test, select the data streams and corresponding parameters that need to be synchronized, integrate the time series of the multi-stream data of the flight test, obtain the timestamp information of the telemetry data stream, flight parameter data stream, and video data stream from the time series database, set the absolute time, select one stream as the reference time stream, and synchronize the remaining multi-streams according to the reference time stream;

[0068] The time-synchronized data is loaded in the visualization control to complete the display. The visualization control supports the display of curves, instruments, and two- and three-dimensional GIS maps. Driven by multi-stream data, the flight process can be replayed in time under the unified control of the time control bar.

[0069] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A flight test multi-stream data time synchronization method, characterized in that: The method comprises the following steps: Step S1: The flight test multi-stream data includes a flight test telemetry data stream, an airborne flight parameter data stream, an airborne audio data stream, and an airborne video data stream, and the flight test multi-stream data is uniformly managed by a flight test multi-stream data time synchronization tool; Step S2: The flight test multi-stream data time synchronization tool uses a time series database to store the flight test telemetry data stream and the airborne flight parameter data stream. The time series database provides time series retrieval capabilities. Step S3: The airborne audio data stream and the airborne video data stream are stored in a file format in a data frame sequence; Step S4: The flight test multi-stream data time synchronization tool uses a CNN-like algorithm to identify the image timestamps of the airborne video data stream on the trained model to obtain a high-precision video image time series; Step S5: Integrate the time series of multi-stream data of the flight test, set the absolute time and the reference time stream, and synchronize the remaining multi-stream data according to the reference time stream; Step S6: After the time synchronization is completed, the flight test process is replayed in the flight test playback display screen in the flight test multi-stream data time synchronization tool; Step S4 specifically includes the following steps: Step S41: pre-processing the airborne video sample picture data set; using a labeling tool to label the timestamp area in the sample picture in the form of text box coordinates; Step S42: training a video timestamp recognition model; (1) The CNN-like algorithm model is composed of CNN+RNN. CNN generates two feature maps: a region score map, which shows the probability of the center area of ​​each character in the image and scores the position of each character in the image; and an affinity score map, which represents the probability of the center of the adjacent character area and is used to evaluate the connection strength between characters and determine which characters are connected to each other. (2) RNN completes the recognition of curved text. By associating the text order, it helps the model understand the text context, completes the prediction of the feature sequence, learns each feature vector in the sequence, and outputs the predicted label, which is the distribution of the true value; Step S43: training a CNN-like algorithm model; Step S44: Loading the CNN-like algorithm model to extract timestamp; The obtained CNN-like algorithm model is placed in the system. Each time the timestamp recognition is performed, the user first defines the timestamp area position, performs image extraction and region segmentation on the video data, performs prediction and recognition on the segmented image, and obtains the timestamp of the data frame. Step S45: establish a corresponding relationship between the data frame and the timestamp, and store them uniformly in the time series database.

2. The method according to claim 1, characterized in that The step S1: the flight test multi-stream data includes the flight test telemetry data stream, the airborne flight parameter data stream, the airborne audio data stream and the airborne video data stream, which are uniformly managed by the flight test multi-stream data time synchronization tool, specifically including: Process telemetry data streams according to the synchronization word, full frame, and subframe of the fixed frame structure, extract processing parameters according to time period or the whole process, and provide range selection for time period according to the time extracted from the frame of the telemetry data stream. The whole process supports the processing of the whole time period and provides parameter selection. Users can select or select all parameters for output; Import, store, process and query flight parameter data stream; wherein, by decoding the flight parameter data stream, the timestamp in the data frame is obtained and the timestamp sequence of the flight parameter data parameters is established; Import, store, process and query airborne audio; airborne audio formats include MP3, WMA, WAV, and encrypted audio data, extract channel and track time length information from audio; Import, store, process and query of airborne videos; airborne videos include H.264 and H.265 airborne video encoding formats. During the processing, the decoding timestamp and display timestamp information are extracted when decoding the video, and used as the basic start and end time information of the video data.

3. The method according to claim 2, characterized in that Step S2: The flight test multi-stream data time synchronization tool uses a time series database to store the flight test telemetry data stream and the airborne flight parameter data stream. The time series database provides time series retrieval capabilities, specifically including: A time series database is used to store the timestamps and parameter values ​​of the flight test telemetry data stream and the airborne flight parameter data stream, a time series two-dimensional table structure is established according to the timestamps and parameter values, and a data index is established on the timestamp; Provides an engineering value data query interface for flight test telemetry data streams and airborne flight parameter data streams, which is used for conditional retrieval by parameters or throughout the entire process to support all time periods.

4. The method according to claim 3, characterized in that: In step S3, the airborne audio data stream and the airborne video data stream are stored in a file format in the order of data frames. Specifically, the airborne audio data stream or the airborne video data stream is stored in a file plus metadata manner. The file is used to store audio and video data, and the metadata is used to provide information extension for the audio and video data, recording the audio and video name, size, format, length, frame start and end time information.

5. The method according to claim 4, characterized in that The specific implementation of step S5 is: Step S51: aligning timestamps in different time formats; Step S52: Setting a reference time stream, comparing the rest of the data streams according to the start and end time, sampling frequency, and timestamp of the reference time stream, and making a secondary adjustment of the absolute time of the reference time; Step S53: setting the start and end time and sampling rate of the remaining aligned time streams; Step S54: Align the multi-stream data of the flight test according to the time alignment algorithm. The time alignment algorithm is as follows: Step S541: Based on the reference time stream, compare the start and end times of the remaining multiple streams to find the start time position and end time position of the remaining data streams to start the comparison; Step S542: Based on the reference time stream, compare the sampling rates of the remaining multiple streams, calculate the sampling rate difference, and obtain the number of points that need to be interpolated; In the above formula, is the sampling rate difference, For high sampling rates, For low sampling rate, after calculating the sampling rate difference, if it is greater than the benchmark time stream sampling rate, then do the jump point calculation, the number of jump points is ; If it is less than the base time stream sampling rate, then interpolation is performed, and the number of interpolation points is , interpolation uses linear / nonlinear interpolation; Step S543: Obtain the timestamp of the reference time stream and compare it with the timestamps of the remaining data streams. Since the flight test data is accurate to milliseconds, it is necessary to compare the time from milliseconds. The alignment method is as follows: (1) The sampling rates of all data streams have reached the same level; (2) Take the starting timestamp of the reference time stream, find the alignment timestamps of all other streams according to the minimum difference method, and replace their timestamps with the timestamp of the reference time stream; (3) After completion, take the next timestamp of the base time stream and follow the method (2) until the end; (4) Finally, aligned multi-stream data is obtained.

6. The method according to claim 5, characterized in that The visualization control of the flight test multi-stream data time synchronization tool supports the display of curves, instruments, and two- and three-dimensional GIS maps. It completes the playback of the flight process under the unified control of the time control bar through the drive of multi-stream data.

7. The method according to claim 1, characterized in that In step S41, the marking method is: (1) Extract key image frames from the video; (2) Delimiting the display area of ​​the timestamp in the key image frame; (3) Label the text information corresponding to the timestamp. The labeled samples should include the normally displayed timestamp and the distorted timestamp to generate a data set.

8. The method according to claim 1, characterized in that In step S42, CNN uses the VGG-16 model, which extracts local features from the airborne video image through convolution kernels, performs downsampling, and preprocesses the input image, including adjusting the image size to ensure that the length and width are multiples of 32, and reducing the dimension of the feature map through a pooling layer, and then upsampling these downsampled feature maps and combining them with the original feature maps.

9. The method according to claim 1, characterized in that: In step S43, the sample images are arranged in the correct time sequence, the relevant parameters of the training CNN algorithm model are configured, and the model training begins.

Images