Multi-source heterogeneous bus data parallel acquisition recording method and system thereof

Abstract

The present application relates to the technical field of data acquisition, in particular to a kind of multi-source heterogeneous bus data parallel acquisition recording method and system thereof, comprising: initialization task cycle and maximum sampling period;Task in task queue is scheduled based on task cycle, time step is set and markable task is marked;According to the number of markable tasks, sampling channel is set and is distributed to sampling thread;Acquisition data is packed as acquisition data packet and is aligned with time stamp as reference;Offset calculation and compensation are carried out to acquisition data packet;According to data capacity and sampling frequency, data acquisition thread is started;The time difference and content similarity of adjacent data packet are detected, and intelligent data merging is executed;Real-time storage is handled data packet;Self-adapting task scheduling and accurate time alignment technology, the efficient parallel acquisition and accurate record of heterogeneous bus data are realized, and the synchronism, integrity and processing efficiency of data acquisition are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a method and system for parallel acquisition and recording of multi-source heterogeneous bus data, applicable to systems in aerospace, equipment testing, industrial automation, and other fields that require simultaneous processing of multiple different types of bus data. Background Technology

[0002] In modern complex control systems, it is often necessary to simultaneously acquire data from multiple different types of buses, such as serial buses, CAN buses, ARINC429 buses, MIL-STD-1553 buses, Ethernet, audio / video buses, analog buses, and digital buses. These buses have different data formats, transmission rates, and timing characteristics, posing numerous challenges when performing parallel data acquisition and storage across multiple buses.

[0003] Traditional data acquisition methods typically employ fixed-frequency sequential sampling, which struggles to adapt to the varying data characteristics of different buses, often resulting in insufficient sampling of high-speed data or oversampling of low-speed data. Furthermore, the lack of an effective time alignment mechanism makes it difficult to guarantee the consistency of data from different buses over time, reducing the value of data analysis and utilization. In addition, traditional methods often couple data acquisition and processing together, limiting the system's parallel processing capabilities and making it difficult to meet the real-time acquisition requirements of large-scale heterogeneous data.

[0004] Therefore, there is an urgent need for a new method that can adapt to the characteristics of multi-source heterogeneous buses and achieve efficient parallel data acquisition and precise time alignment. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for parallel acquisition and recording of multi-source heterogeneous bus data, which aims to solve the problems of poor synchronization of heterogeneous bus data acquisition, unreasonable resource utilization, and difficulty in ensuring data integrity in the prior art.

[0006] Specifically, this invention aims to achieve the following objectives: construct an adaptive task scheduling mechanism to dynamically adjust the acquisition strategy according to the characteristics of different bus data, thereby improving system resource utilization efficiency; design a precise time alignment technology to ensure the consistency of data from different buses in the time dimension; achieve decoupling of data acquisition and processing, and improve system throughput through multi-threaded parallel processing; establish a data integrity verification mechanism to promptly detect and handle data loss; and optimize data storage strategies to reduce redundant data and improve storage and subsequent processing efficiency.

[0007] This invention proposes a method for parallel acquisition and recording of multi-source heterogeneous bus data, comprising:

[0008] Initialize the task cycle and maximum sampling cycle, and create a task queue containing multiple bus data acquisition tasks of different types;

[0009] Based on the task cycle, the tasks in the task queue are scheduled, a time step is set, and schedulable tasks are marked at specific times.

[0010] Based on the number of schedulable tasks, a corresponding number of sampling channels are set, and each sampling channel is assigned to a sampling thread for the collection of corresponding data types.

[0011] The collected data is packaged into a data collection package and aligned with the data collection package based on the timestamp.

[0012] Offset calculation and compensation are performed on the collected data packets to ensure data integrity;

[0013] The timing of starting the data acquisition thread is controlled based on the data capacity and sampling frequency conditions.

[0014] Detect the time difference and content similarity of adjacent collected data packets, and perform intelligent data merging;

[0015] The processed data packets are stored in real time.

[0016] Preferably, the steps of initializing the task period and the maximum sampling period specifically include:

[0017] Obtain the period value of each task in the task queue;

[0018] Calculate the greatest common divisor of the periodic values;

[0019] Determine whether the greatest common divisor is less than the maximum data acquisition speed supported by the system;

[0020] When the greatest common divisor is less than the maximum data acquisition speed, the greatest common divisor is set as the task period, and the maximum sampling period is set to zero.

[0021] When the greatest common divisor is greater than or equal to the maximum data acquisition rate, the task cycle and the maximum sampling cycle are determined by an iterative optimization algorithm.

[0022] Preferably, the step of scheduling tasks in the task queue specifically includes:

[0023] Determine the status of each task and classify the task status into three categories: idle, started, and in progress.

[0024] Record the current time of the ongoing task and set the remaining time for the ongoing task;

[0025] Determine whether a task in progress is marked as idle within its own cycle, and set any task that is not marked as idle as a valid task;

[0026] Assign valid ongoing tasks to the main thread for execution;

[0027] Monitor the execution status of tasks, and mark an active task as idle when it reaches the end of its cycle.

[0028] When the remaining time is less than zero, the corresponding in-process task is marked as idle.

[0029] Preferably, the acquisition process of the sampling channel specifically includes:

[0030] The main thread monitors each main channel in real time and captures data acquisition signals.

[0031] The collected data is written to the data cache queue;

[0032] Monitor the capacity of the data in the data cache queue and the current time;

[0033] When the capacity of data in the data cache queue exceeds the preset capacity threshold, or when the current time reaches the preset sampling period, the data acquisition thread is started.

[0034] The data acquisition thread reads data from the data cache queue and packages it into an acquisition data packet.

[0035] Check if the tags of the collected data packets are abnormal; if abnormal, repackage them.

[0036] Normal data packets are written to the storage buffer and sent to the data receiving module.

[0037] Preferably, the step of aligning the collected data packets based on the timestamp specifically includes:

[0038] Obtain the timestamp of the currently collected data and calculate the time interval between it and the initial timestamp;

[0039] The acquired data packets are split into one or more aligned units;

[0040] Align the beginning of the data in the alignment unit with the initial timestamp;

[0041] Add each alignment unit to the alignment queue;

[0042] The alignment units are sorted in chronological order in the alignment queue;

[0043] Process the first alignment unit and determine whether its capacity meets the preset conditions;

[0044] When the capacity of the first alignment unit meets the condition, it is written into the storage buffer.

[0045] Preferably, the step of calculating and compensating the offset of the acquired data packet specifically includes:

[0046] When data needs time alignment and the current time is not within the preset sampling period:

[0047] Calculate the difference between the beginning of the data in each alignment unit and the corresponding timestamp;

[0048] The alignment unit with the smallest absolute difference is identified as the time alignment unit, and the rest are identified as data alignment units.

[0049] Determine if there are any alignment units with a difference of zero;

[0050] When there is no alignment unit with a difference of zero, record the data offset in the time alignment unit;

[0051] The data offsets are set in each data alignment unit according to the alignment order;

[0052] Construct a time-aligned data packet containing complete alignment information and write it to the storage buffer.

[0053] Preferably, the timestamp update method specifically includes:

[0054] Monitor the amount of data in the data cache queue;

[0055] When the amount of data is less than one-third of the task cycle, calculate the time interval between the current time and the last updated timestamp.

[0056] Determine whether the time interval is greater than one-third of the task cycle;

[0057] When the time interval is greater than one-third of the task cycle, the current time is set as the updated timestamp;

[0058] Use the updated timestamp as the benchmark for subsequent data alignment.

[0059] Preferably, the step of performing intelligent data merging specifically includes:

[0060] The capacity of the data buffer queue for monitoring time-aligned data packets;

[0061] When the capacity is greater than the preset threshold and the current time is not within the preset sampling period, the timestamp is updated to the current time.

[0062] Read the two sampled data packets that are closest in time from the storage buffer, and use them as the first data packet and the second data packet, respectively;

[0063] Calculate the time difference between the first data packet and the second data packet;

[0064] Compare the content similarity of the aligned first and second data packets;

[0065] When the time difference is less than a preset time difference threshold and the contents of the two data packets are highly similar, data merging is performed;

[0066] The repackaged and merged data is written to the data cache queue.

[0067] Preferably, the step of verifying the integrity of the collected data is also included:

[0068] Calculate the theoretical sampled data count based on the sampling frequency of the sampling channel;

[0069] Record the actual number of data collected;

[0070] Compare the difference between the theoretical sampled data count and the actual data count;

[0071] When the difference is greater than zero, a packet loss alarm is generated;

[0072] When the difference is equal to zero, record the start and end times of writing data into the buffer.

[0073] Calculate the time difference between the start and end times and use it as the timestamp of the data.

[0074] Preferably, the method for processing alignment units specifically includes:

[0075] Get the number of data items in the alignment buffer;

[0076] Different processing paths are selected based on the amount of data and timestamp information;

[0077] When the alignment buffer is empty or the amount of data is less than a preset threshold and there is no alignment unit with a timestamp of zero, the data in the data cache queue is aligned according to the data and timestamp.

[0078] If the alignment buffer is empty, proceed directly to the final processing step;

[0079] When the amount of data is less than the preset threshold and there is an alignment unit with a timestamp of zero, clear the alignment unit;

[0080] When the amount of data is greater than or equal to a preset threshold, the data in the data cache queue is split into individual alignment units and processed in alignment order;

[0081] The processed alignment units are written to the alignment buffer in timestamp order.

[0082] A multi-source heterogeneous bus data parallel acquisition and recording system for implementing the method includes: a central control computer for acquisition and storage, used for unified configuration and management of heterogeneous bus data acquisition tasks; multiple heterogeneous bus interface acquisition modules, each module corresponding to process one or more types of bus data; the central control computer is connected to each heterogeneous bus interface acquisition module via PoE high-speed Ethernet to realize software-defined configuration of acquisition module parameters, acquisition timing control, and transmission and collection of acquired data; wherein, the central control computer runs the multi-source heterogeneous bus data parallel acquisition and recording method, and each heterogeneous bus interface acquisition module executes a corresponding data acquisition task;

[0083] The data normalization processing module is used to convert heterogeneous data collected from different bus types into a unified data format;

[0084] The data normalization processing module adds a unified header structure to each acquired data packet, which includes bus type identifier, data source identifier, timestamp information, data length, and verification information.

[0085] The unified data acquisition data packets are transmitted and stored within the system through a standardized transmission protocol for subsequent data processing and analysis.

[0086] A high-capacity data storage module is used to store processed acquisition data packets in a high-speed streaming manner;

[0087] The data index management module is used to create multidimensional indexes based on timestamps and bus types, supporting efficient data retrieval after experiments;

[0088] The system supports correlation analysis and reconstruction of multi-source heterogeneous bus data based on the index after the experiment, realizing the post-processing analysis of the experiment data.

[0089] The multi-source heterogeneous bus data parallel acquisition and recording method provided by this invention has the following beneficial effects:

[0090] 1. Through an adaptive task scheduling mechanism, the system can dynamically allocate resources according to the characteristics of different bus data, which significantly improves resource utilization efficiency. Compared with the traditional fixed-period acquisition method, the resource utilization rate is increased by more than 30%.

[0091] 2. The innovative time alignment technology enables precise time synchronization of heterogeneous data, with a time alignment accuracy down to the microsecond level, providing a reliable foundation for subsequent data analysis;

[0092] 3. The multi-threaded parallel processing architecture enables the system to process information from multiple data sources simultaneously, significantly improving data processing throughput. In actual tests, it can process data from more than 10 different types of buses simultaneously.

[0093] 4. The combination of the minimum difference strategy and offset compensation technology ensures data integrity and accuracy, and significantly reduces information loss during the data reconstruction process;

[0094] 5. Intelligent data merging technology effectively identifies and processes redundant data, reducing storage space usage by more than 50% while ensuring data integrity;

[0095] 6. A complete anomaly detection and handling mechanism significantly improves system stability and reliability, making it suitable for long-term stable operation in complex industrial environments;

[0096] 7. It adopts a hardware PoE master-slave architecture and a software-defined hardware interface configuration design, which facilitates the splitting and combination of the acquisition system and increases the adaptability of acquisition devices and the convenience of multi-task switching. Attached Figure Description

[0097] Figure 1 This is an overall flowchart of the method of the present invention;

[0098] Figure 2 Initialize the flowchart for the task cycle;

[0099] Figure 3 A flowchart of an adaptive task scheduling mechanism;

[0100] Figure 4 This is a flowchart of the data acquisition and processing process;

[0101] Figure 5 Flowchart for time alignment processing;

[0102] Figure 6 Flowchart for timestamp update method;

[0103] Figure 7 Flowchart for data integrity verification;

[0104] Figure 8 This is a flowchart of the intelligent data merging process. Detailed Implementation

[0105] Please refer to Figures 1-8 The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that these embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0106] Reference Figure 1 The method for parallel acquisition and recording of multi-source heterogeneous bus data provided by this invention mainly includes the following steps:

[0107] First, the task cycle and maximum sampling cycle are initialized, and a task queue containing multiple different types of bus data acquisition tasks is created. In one embodiment of the present invention, the task queue may contain various types of data acquisition tasks such as CAN bus, ARINC429 bus, and MIL-STD-1553 bus.

[0108] Next, based on the task cycle, tasks in the task queue are scheduled, a time step is set, and schedulable tasks are marked at specific times. Preferably, the system marks tasks in progress as schedulable tasks at time Tk·Δt (where T is the task cycle, Δt is the time step, and k is an integer greater than 0).

[0109] Based on the number of schedulable tasks, the system sets up a corresponding number of sampling channels, with each sampling channel assigned to a sampling thread for collecting the corresponding type of data. This dynamic resource allocation mechanism can flexibly adjust system resources according to actual needs, significantly improving resource utilization efficiency.

[0110] During data acquisition, the system packages the collected data into acquisition data packets and aligns these packets using timestamps as a reference. Simultaneously, offset calculations and compensations are performed on the acquisition data packets to ensure data integrity. This process is crucial for ensuring the consistency of data from different buses across the time dimension.

[0111] In addition, the system controls the start time of the data acquisition thread based on the data capacity and sampling frequency conditions, thereby decoupling data acquisition and processing and improving the system's parallel processing capability.

[0112] The system also detects the time difference and content similarity of adjacent data packets, performs intelligent data merging, reduces redundant data, and improves storage and processing efficiency. Finally, the system stores the processed data packets in real time, completing the entire data acquisition and recording process.

[0113] The key steps of the method of the present invention will be described in detail below.

[0114] Reference Figure 2 The initialization process of the task period and the maximum sampling period in the method of this invention is as follows:

[0115] First, the system acquires the period values ​​of each task in the task queue. These period values ​​are usually preset by the user based on the characteristics of different buses and data acquisition requirements. In a typical application scenario, the acquisition period of a high-speed CAN bus may be set to 1 millisecond, while the acquisition period of a low-speed MIL-STD-1553 bus may be set to 10 milliseconds.

[0116] Next, the system calculates the greatest common divisor (GCD) of these periodic values. In one embodiment of the invention, the GCD is calculated recursively:

[0117] GCD(a,b)=GCD(b,amodb), when b!=0

[0118] GCD(a,0)=a

[0119] The greatest common divisor of multiple task cycles is calculated as follows:

[0120] GcdN=GCD(GCD(T1,GCD(T2...GCD(Tn))))

[0121] Where T1, T2, ..., Tn represent the period values ​​of each task.

[0122] After calculating the greatest common divisor, the system will determine whether the value is less than the maximum data acquisition speed supported by the system. Preferably, the maximum data acquisition speed is usually determined by hardware conditions. For example, in a typical industrial control system, this speed may be 100kHz (i.e., the minimum period is 10 microseconds).

[0123] When the greatest common divisor is less than the maximum data acquisition rate, the system directly sets the greatest common divisor to the task period (T=GcdN) and sets the maximum sampling period to zero (T0=0). In this case, all tasks can be executed at their optimal period, and the system resource utilization is optimized.

[0124] However, when the greatest common divisor (GCD) is greater than or equal to the maximum data acquisition rate, directly using the GCD as the task period will exceed the system's processing capacity. In this case, the system will determine a suitable task period and maximum sampling period through an iterative optimization algorithm. The specific process is as follows:

[0125] 1. Initial settings: T'=T0'=GcdN;

[0126] 2. Set T' to the maximum task cycle value in the task queue;

[0127] 3. Set the task period that has not reached T' to T', and set the initial state to Not Started;

[0128] 4. Determine if T' is greater than the maximum acquisition speed vmax;

[0129] If T'≤vmax, then set T=T' and T0=0;

[0130] If T'>vmax and the number of tasks n=1, then set T=T' and T0=0;

[0131] If T'>vmax and the number of tasks n>1, then set T0'=T', n'=n-1, set the first n'-1 task cycles to T0', and continue to execute the above steps;

[0132] Through this optimization algorithm, the system can preserve the original cycle characteristics of each task as much as possible without exceeding the hardware processing capacity, thereby achieving reasonable allocation and utilization of resources.

[0133] Reference Figure 3 The task scheduling mechanism in the method of this invention is based on a three-state task state model, and is specifically implemented as follows:

[0134] First, the system determines the status of each task in the task queue, classifying them into three categories: idle, started, and in progress. When any task in the task queue begins, it is marked as in progress, while the remaining tasks are marked as idle. Here, "started" indicates that the task's start time has been reached.

[0135] Secondly, the system records the current time of in-process tasks and sets the remaining time for each task. Tasks marked as idle within their own time period T are set as invalid tasks, while valid in-process tasks start at the current time. Here, the initial start time refers to the moment when a task initially enters the in-process state.

[0136] Next, the system assigns valid ongoing tasks to the main thread in the task queue for execution. During execution, the system continuously monitors the task status: if a valid ongoing task finishes its own period T, it is marked as idle; otherwise, the system calculates the remaining time as the current remaining time minus T, and marks valid ongoing tasks with a remaining time less than 0 as idle.

[0137] The system repeats the above process until all tasks are idle. This fine-grained task state management and scheduling mechanism enables the system to precisely control the execution timing and resource allocation of each task, significantly improving the system's task processing efficiency and resource utilization.

[0138] Preferably, in one embodiment of the present invention, if the system detects a new task in the task queue, the status of the new task is set to idle, with an initial period of "not started" and an initial start time of the current time. This dynamic task addition mechanism enables the system to flexibly respond to changing data acquisition needs.

[0139] Reference Figure 4 The specific implementation of the sampling channel acquisition process in the method of this invention is as follows:

[0140] First, the main thread monitors each main channel in real time, performing a data acquisition operation each time a data acquisition signal is detected. In industrial environments, data acquisition signals are typically triggered by hardware interrupts or software timers, with the signal frequency depending on the bus type and configuration. For example, a high-speed CAN bus might trigger an acquisition signal every 1 millisecond, while a low-speed bus might have a lower frequency.

[0141] Once data is collected, the system writes it to a data cache queue. The data cache queue uses a first-in, first-out (FIFO) structure to temporarily store the collected raw data. Preferably, the size of the cache queue can be dynamically adjusted according to system memory resources and data processing speed, with a typical value of 1MB to 10MB.

[0142] Next, the system monitors the capacity of the data in the data cache queue and the current time to determine when to start the data acquisition thread. Specifically, the system judges the following two conditions:

[0143] 1. Does the capacity of the data in the data cache queue exceed the preset capacity threshold?

[0144] 2. Has the preset sampling period been reached at the current moment?

[0145] When any condition is met, the system starts the data acquisition thread. Preferably, the capacity threshold can be set to 70% to 80% of the buffer queue capacity, while the sampling period depends on the data analysis requirements, typically ranging from 10 milliseconds to 100 milliseconds.

[0146] After the data acquisition thread starts, it reads data from the data cache queue and packages it into acquisition data packets. Each acquisition data packet contains information such as raw data, timestamp, and data source identifier, providing a complete context for subsequent processing.

[0147] Before writing the acquired data packets to the storage buffer, the system checks whether the packet tags are abnormal. Abnormalities include, but are not limited to, data checksum errors, format mismatches, and abnormal lengths. If an abnormality is detected, the system repackages the data packets; if no abnormality is detected, the data packets are written to the storage buffer and sent to the data receiving module.

[0148] This data acquisition mechanism, based on dual-condition triggering, enables the system to balance real-time performance and processing efficiency. It can process accumulated data in a timely manner and provide data analysis results according to a fixed period, meeting the needs of different application scenarios.

[0149] Reference Figure 5 The process of aligning the collected data packets based on timestamps in the method of this invention is as follows:

[0150] First, the system obtains the timestamp of the currently collected data and calculates the time interval between it and the initial timestamp. Here, the initial timestamp refers to the timestamp of the first data item in the data cache queue, serving as the reference point for subsequent alignment. In one embodiment of the invention, the timestamp accuracy can reach the microsecond level, ensuring high-precision time alignment.

[0151] Next, the system splits the acquired data packets into one or more alignment units. An alignment unit is the basic processing unit for data alignment, and its size depends on the data characteristics and alignment accuracy requirements. Preferably, the size of the alignment unit can be configured to be 10% to 25% of the original data packet size, balancing alignment accuracy and processing overhead.

[0152] The system aligns the beginning of the data in the alignment unit with the initial timestamp, establishing a time mapping relationship. This step ensures that data from different sources can be compared and analyzed on a unified time base.

[0153] The system then adds each alignment unit to the alignment queue and sorts them in chronological order. The alignment queue is a priority queue sorted by timestamps to ensure that data is processed in chronological order and to maintain the temporal integrity of the data.

[0154] The system processes the first alignment unit in the alignment queue (referred to as the first alignment unit) and determines whether its capacity meets preset conditions. Specifically, the system checks the following conditions:

[0155] 1. Whether the capacity of the first alignment unit is less than the preset alignment capacity threshold;

[0156] 2. Check if the capacity of the aligned queue is not less than 1;

[0157] When the capacity of the first alignment unit is less than the threshold and the queue capacity is not less than 1, the system removes the first alignment unit from the queue and waits for more data to accumulate before processing. Preferably, the alignment capacity threshold can be set to 15% to 20% of the complete data packet size to ensure that the processed data units are sufficiently meaningful.

[0158] When the capacity of the first alignment unit is not less than the preset alignment capacity threshold, the system writes it into the storage buffer and enters the subsequent processing flow as aligned data.

[0159] This timestamp-based precise alignment mechanism ensures the consistency of data from different buses in the time dimension, providing a reliable foundation for subsequent data analysis and processing.

[0160] The process of offset calculation and compensation for the acquired data packets in the method of this invention is as follows:

[0161] When data requires time alignment and the current time is not within the preset sampling period, the system first splits the data into multiple alignment units, and then calculates the difference between the beginning of the data in each alignment unit and the corresponding timestamp. This difference reflects the deviation between the actual sampling time and the ideal sampling time, and is the basis for subsequent compensation.

[0162] The system identifies the alignment unit with the smallest absolute difference as the time alignment unit, and marks the other units as data alignment units. Preferably, the difference is calculated using the following method:

[0163] Difference = |Tstart - Tideal|

[0164] Where Tstart is the start timestamp of the alignment unit, and Tideal is the closest ideal sampling time point.

[0165] In practical applications, the ideal sampling time is usually determined based on a fixed sampling period, such as one sampling point every 10 milliseconds. However, the actual sampling time is affected by various factors, such as system load and interrupt latency, resulting in deviations from the ideal time. By identifying the unit with the smallest difference as the time reference, the system can minimize time alignment errors.

[0166] Next, the system determines whether there is an alignment unit with a difference of zero. A difference of zero means that the sampling time of this unit falls exactly at the ideal sampling point, which is the most ideal time alignment reference.

[0167] When no alignment unit with a difference of zero exists, the system records the data offset in the time alignment unit and sets this offset in each data alignment unit according to the alignment order. Preferably, the offset is recorded in a relative offset format, that is, an offset relative to the position of the time alignment unit, rather than an absolute position, to reduce storage overhead.

[0168] Finally, the system constructs a time-aligned data packet containing complete alignment information and writes it to the storage buffer. This data packet contains information such as the original data, timestamps, and offsets, providing a comprehensive basis for subsequent data reconstruction and analysis.

[0169] This offset calculation and compensation technology solves the problem of time asynchrony in heterogeneous data acquisition, ensures the temporal consistency and integrity of data, and greatly improves the accuracy and reliability of multi-source data analysis.

[0170] Reference Figure 6 The timestamp update method in this invention is as follows:

[0171] First, the system continuously monitors the amount of data in the data cache queue. Changes in the amount of data reflect the dynamic balance between data generation and processing, and are an important basis for determining the timestamp update frequency.

[0172] When the amount of data in the data cache queue is less than one-third of the task cycle, the system calculates the time interval between the current moment and the last timestamp update. This condition indicates that the data processing speed exceeds the data generation speed, and the data in the queue is insufficient. In this case, it may be necessary to update the timestamp to maintain the system's time accuracy.

[0173] The system determines whether the calculated time interval is greater than one-third of the task cycle. This threshold setting is based on experimental verification and achieves a good balance between maintaining timestamp accuracy and reducing update overhead. Preferably, this threshold can be adjusted according to actual needs for different application scenarios; for example, it can be set to one-quarter of the task cycle for scenarios with high time accuracy requirements.

[0174] When the time interval exceeds one-third of the task cycle, the system sets the current time as the updated timestamp. This dynamic update strategy based on data volume and time interval ensures the accuracy of the timestamp while avoiding the system overhead caused by frequent updates, providing a reliable time benchmark for the accurate alignment of heterogeneous data.

[0175] In one embodiment of the present invention, timestamp updates can also be adjusted in conjunction with system load status. When the system load is high, the update frequency can be appropriately reduced; when the load is low, the update frequency can be increased to further optimize system performance. Specifically, a dynamically adjusted threshold can be obtained by multiplying a standard threshold (one-third of the task cycle) by a system load coefficient. The system load coefficient typically ranges from 0.8 to 1.2; the higher the load, the larger the coefficient, and the lower the update frequency.

[0176] The updated timestamp serves as the benchmark for subsequent data alignment and plays a crucial role in ensuring system time consistency. Through this intelligent update strategy, the system can maintain optimal performance under various operating conditions.

[0177] Reference Figure 8 The intelligent data merging process in the method of this invention is as follows:

[0178] First, the system monitors the data buffer queue capacity of the time-aligned data packets. When the capacity exceeds a preset threshold and the current time is not within the preset sampling period, the system will consider performing a data merging operation. Preferably, the capacity threshold can be set to 75% to 85% of the maximum queue capacity, ensuring timely processing while providing sufficient data for similarity analysis.

[0179] At this point, the system updates the timestamp to the current moment to reflect the latest data status. Then, it reads the two most recently sampled data packets from the storage buffer, designating them as the first and second data packets, respectively. These two data packets are potential candidates for merging.

[0180] Next, the system calculates the time difference between the first and second data packets and compares their content similarity. The time difference is calculated as the absolute difference between the timestamps of the two data packets, while the content similarity is determined by comparing the values ​​of the data fields.

[0181] In one embodiment of the present invention, the time difference threshold can be set to 10% to 20% of the sampling period. For example, when the sampling period is 100 milliseconds, the time difference threshold can be set to 10 to 20 milliseconds. Content similarity determination can be based on the following methods:

[0182] Similarity = Number of identical fields / Total number of fields;

[0183] When the similarity exceeds a preset threshold (e.g., 85% to 95%), the two data packets are considered to be highly similar.

[0184] When the time difference is less than a preset time difference threshold and the contents of two data packets are highly similar, the system performs a data merging operation. The merging strategy typically involves retaining the data packet with the more recent timestamp and discarding the older data packet, or merging the unique information of the two to form a new data packet.

[0185] Finally, the system repackages the merged data and writes it to the data cache queue, awaiting further processing. This intelligent merging technology effectively reduces redundant data, lowers storage and processing overhead, while maintaining the integrity of critical data, making it particularly suitable for processing slowly changing signal data.

[0186] In practical applications, the system can employ different merging strategies based on the characteristics of different data types. For example, for state-changing data, only the data at the state change points can be retained; for continuously changing data, sampling retention or interpolation compression can be used for merging. This flexible strategy enables the system to adapt to various data types and optimize storage and processing efficiency.

[0187] Reference Figure 7 The specific steps for verifying the integrity of the collected data in the method of this invention are as follows:

[0188] First, the system calculates the theoretical sampled data count based on the sampling frequency of the sampling channel. The sampling frequency is a fundamental attribute of the sampling channel and varies depending on the bus type and configuration. The theoretical sampled data count is calculated as follows:

[0189] Theoretical count = sampling period × sampling frequency

[0190] The sampling period refers to the time interval between the start of data collection and the checkpoint, usually measured in seconds; the sampling frequency is measured in Hertz (Hz), representing the number of samples per second.

[0191] For example, for a channel with a sampling frequency of 1kHz and a sampling period of 0.5 seconds, theoretically 500 data points should be collected.

[0192] Next, the system records the actual data count collected and compares the difference between the theoretical count and the actual count. The difference is calculated as follows:

[0193] Difference = Theoretical count - Actual count;

[0194] When the difference is greater than zero, it indicates that data loss has occurred, and the system will generate a packet loss alarm. The packet loss alarm typically includes information such as the number of lost packets, the time period, and the channel identifier, which facilitates subsequent analysis and processing.

[0195] In one embodiment of the present invention, the system can also set different levels of alarms for different packet loss rates. For example, a minor warning is generated when the packet loss rate is below 1%, a general alarm is generated when the rate is between 1% and 5%, and a severe alarm is generated when the rate exceeds 5%. The packet loss rate is calculated as follows:

[0196] Packet loss rate = (Difference / Theoretical count) × 100%;

[0197] When the difference is zero, it indicates that the data is complete and without loss. The system will record the start and end times of the data being written to the buffer. These two time points are important bases for evaluating the temporal integrity of the data.

[0198] The system calculates the time difference between the start and end times and uses it as the data's timestamp. This timestamp reflects the actual time span of data collection, providing an accurate time reference for subsequent data alignment and analysis.

[0199] The integrity verification mechanism enables the system to detect and handle data loss in a timely manner, greatly improving the reliability of data acquisition and the credibility of results. It is particularly suitable for application scenarios with high requirements for data integrity, such as aerospace, industrial automation and other fields.

[0200] The specific processing method for the alignment unit in the method of this invention is as follows:

[0201] First, the system retrieves the amount of data in the alignment buffer. The alignment buffer is a temporary storage area for data to be aligned, and its size reflects the current accumulation of data to be processed.

[0202] Based on the amount of data in the cache and the timestamp information, the system selects different processing paths. The system judges the following conditions:

[0203] 1. Check if the alignment buffer is empty (data count m=0);

[0204] 2. Whether the number of data is less than the preset threshold (m < threshold) and whether there are no alignment units with timestamps of 0;

[0205] 3. Whether the amount of data is less than the preset threshold and whether there are alignment units with timestamps of 0;

[0206] 4. Whether the number of data points is greater than or equal to the preset threshold (m≥threshold);

[0207] Preferably, the preset threshold can be set according to the system's processing capacity and real-time requirements, typically ranging from 10 to 50 alignment units. This threshold balances batch processing efficiency and data processing latency, achieving a good balance between processing efficiency and real-time performance.

[0208] When the alignment buffer is empty or the amount of data is less than a preset threshold and there are no alignment units with a timestamp of 0, the system aligns the data in the data cache queue according to the data and timestamp. In this case, the amount of data to be processed is small and can be processed directly without complex cache management.

[0209] When the alignment buffer is empty, the system jumps directly to the final processing step. At this point, there is no data to be processed, and the system can wait for new data to arrive.

[0210] When the amount of data is less than a preset threshold and there are alignment units with a timestamp of 0, the system clears the alignment unit. Alignment units with a timestamp of 0 usually indicate missing or abnormal time information, making effective alignment impossible and requiring clearing to avoid affecting other data processing.

[0211] When the amount of data is greater than or equal to a preset threshold, the system splits the data in the data cache queue into individual aligned units and processes them in alignment order. In this case, when enough data has accumulated, batch processing can be performed to improve efficiency.

[0212] Finally, the system writes the processed alignment units into the alignment buffer in timestamp order. This timestamp order ensures the temporal integrity of the data processing, providing a reliable foundation for subsequent analysis.

[0213] In one embodiment of the present invention, the system can also adopt different processing strategies for the alignment units according to the characteristics of different types of data. For example, for high-priority data, a smaller processing latency can be set; for low-priority data, the cache size can be appropriately increased to improve batch processing efficiency. This flexible strategy enables the system to be optimized for different application needs and provide more efficient data processing services.

[0214] Through the detailed description of the above embodiments, those skilled in the art should be able to understand and implement the multi-source heterogeneous bus data parallel acquisition and recording method of the present invention. The present invention solves the problems of synchronization, integrity, and efficiency in heterogeneous bus data acquisition through innovative technologies such as task scheduling, time alignment, data processing, and integrity verification, providing a reliable data acquisition solution for fields such as industrial automation and aerospace.

[0215] In addition to the aforementioned method for parallel acquisition and recording of multi-source heterogeneous bus data, this invention also provides a hardware system architecture for implementing this method. The hardware system adopts a top-down architecture design, including:

[0216] The central control computer for data acquisition and storage serves as the system's central control unit, responsible for the unified configuration and management of heterogeneous bus data acquisition tasks, and executing the aforementioned multi-source heterogeneous bus data parallel acquisition and recording method; multiple heterogeneous bus interface acquisition modules, each module specifically corresponding to process one or more types of bus data, such as CAN bus acquisition module, ARINC429 bus acquisition module, MIL-STD-1553 bus acquisition module, etc.

[0217] In a preferred embodiment of the present invention, the central control computer for data acquisition and storage is connected to each heterogeneous bus interface acquisition module via a PoE high-speed Ethernet connection, enabling software-defined configuration of acquisition module parameters, timing control of acquisition data acquisition, and high-speed transmission and collection of acquired data. This connection method not only provides a data transmission channel but also provides power to each acquisition module through PoE technology, simplifying system wiring and improving system integration and reliability.

[0218] The data normalization processing module of this invention is located within the central control computer for data acquisition and storage, and is used to convert heterogeneous data acquired from different bus types into a unified data format. Specifically, the data normalization processing module adds a unified header structure to each acquired data packet, including a bus type identifier (used to distinguish the bus type from which the data originates), a data source identifier (used to locate the specific acquisition module), timestamp information (with microsecond-level precision), data length, and verification information (to ensure data integrity).

[0219] Preferably, the acquisition data packets with a unified data format are transmitted and stored within the system through a standardized transmission protocol. This transmission protocol adopts a lightweight design and includes fields such as frame header (0xAA55), packet length, packet type, source address, destination address, sequence number, data payload, and checksum to ensure the reliability and integrity of data transmission.

[0220] The invention also includes a large-capacity data storage module and a data index management module. The large-capacity data storage module adopts a streaming storage architecture, which can flexibly select a high-speed solid-state drive array or a distributed storage system according to the system configuration, supports continuous write speeds of hundreds of MB / s or even GB / s, and meets the high-speed parallel storage requirements of multi-source heterogeneous bus data.

[0221] The data index management module constructs a multi-dimensional index structure based on timestamps and bus types, supporting the following indexing methods:

[0222] Time-based indexing: Quickly locate data based on timestamp range;

[0223] Bus type index: Filter data by bus type;

[0224] Data Feature Indexing: Indexing based on data content features;

[0225] Composite condition index: Supports queries with multiple combined conditions.

[0226] This multi-dimensional indexing mechanism significantly improves the efficiency of post-experiment data retrieval, making time-synchronized multi-source data correlation analysis highly efficient and feasible. The system supports correlation analysis and reconstruction of multi-source heterogeneous bus data based on the index after the experiment, enabling post-experimental data processing and analysis, and providing comprehensive and reliable data support for equipment testing.

[0227] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-source heterogeneous bus data parallel acquisition recording method, characterized in that, The method comprises the following steps: initializing a task period and a maximum sampling period, creating a task queue comprising a plurality of bus data collection tasks of different types; scheduling the tasks in the task queue based on the task period, setting a time step, and marking a schedulable task at a specific time; setting a corresponding number of sampling channels according to the number of the schedulable tasks, each sampling channel being assigned to a sampling thread for collection of data of a corresponding type; packing the collected data into a collection data packet and aligning the collection data packet based on a time stamp; calculating and compensating for an offset of the collection data packet to ensure data integrity; controlling a start timing of a data collection thread according to data capacity and sampling frequency conditions; detecting a time difference and content similarity of adjacent collection data packets and performing intelligent data merging; storing the processed collection data packet in real time; The step of initializing the task period and the maximum sampling period specifically comprises: obtaining a period value of each task in the task queue; calculating a greatest common divisor of the period values; determining whether the greatest common divisor is less than a maximum data collection speed supported by a system; when the greatest common divisor is less than the maximum data collection speed, setting the greatest common divisor as the task period and setting the maximum sampling period as zero; when the greatest common divisor is greater than or equal to the maximum data collection speed, determining the task period and the maximum sampling period through an iterative optimization algorithm; The step of scheduling the tasks in the task queue specifically comprises: determining a state of each task, dividing the task state into three categories of idle, start and in-process; recording a current time of an in-process task, setting a remaining time of the in-process task; determining whether the in-process task is marked as idle within its own period, setting a task that is not marked as idle as a valid task; assigning the valid in-process task to a main thread for execution; monitoring a task execution state, and when the valid in-process task is executed to the end of the period, marking it as idle; when the remaining time is less than zero, marking the corresponding in-process task as idle; The collection process of the sampling channel specifically comprises: the main thread listens to each main channel in real time, captures a data collection signal; writes the collected data into a data cache queue; monitors a capacity of data in the data cache queue and a current time; when the capacity of data in the data cache queue is greater than a preset capacity threshold or the current time reaches a preset sampling period, starts a data collection thread; the data collection thread reads the data in the data cache queue, packs the data into a collection data packet; checks whether a mark of the collection data packet is abnormal, and repacks if it is abnormal; writes the normal collection data packet into a storage buffer and sends it to a data receiving module; The step of aligning the collection data packets based on the timestamps specifically comprises: obtaining a timestamp of current collection data, calculating a time interval from an initial timestamp; splitting the collection data packets into one or more alignment units; corresponding the data beginning of the alignment units with the initial timestamp; adding the alignment units to an alignment queue; sorting the alignment units in the alignment queue in chronological order; processing a first alignment unit, judging whether its capacity meets a preset condition; when the capacity of the first alignment unit meets the condition, writing it into a storage buffer; The step of calculating and compensating the offset of the collection data packets specifically comprises: when the data needs to be time-aligned and the current time is not a preset sampling period: calculating a difference between the data beginning of each alignment unit and the corresponding timestamp; identifying an alignment unit with the smallest absolute value of the difference as a time-aligned unit, and the rest as data-aligned units; judging whether there is an alignment unit with a difference of zero; when there is no alignment unit with a difference of zero, recording a data offset in the time-aligned unit; setting the data offset in each data-aligned unit according to the alignment order; constructing a time-aligned data packet containing complete alignment information, and writing it into the storage buffer; The updating method of the timestamp specifically comprises: monitoring the data amount in the data cache queue; when the data amount is less than one-third of the task period, calculating a time interval between the current time and the time when the timestamp was last updated; judging whether the time interval is greater than one-third of the task period; when the time interval is greater than one-third of the task period, setting the current time as the updated timestamp; using the updated timestamp as a reference for subsequent data alignment; The step of performing intelligent data merging specifically comprises: monitoring the capacity of the data cache queue of the time-aligned data packet; when the capacity is greater than a preset threshold and the current time is not a preset sampling period, updating the timestamp to the current time; reading two sampling data packets closest in time from the storage buffer as a first data packet and a second data packet respectively; calculating the time difference between the first data packet and the second data packet; comparing the content similarity of the first data packet and the second data packet after alignment; when the time difference is less than a preset time difference threshold, and the content of the two data packets is highly similar, performing data merging; repacking the merged data and writing it into the data cache queue; Also includes the step of integrity check of the collected data: according to the sampling frequency of the sampling channel, the theoretical sampling data count is calculated; the actual data count collected is recorded; the difference between the theoretical sampling data count and the actual data count is compared; when the difference is greater than zero, the packet loss alarm information is generated; when the difference is equal to zero, the start time and the end time of the data write buffer area are recorded; the time difference between the start time and the end time is calculated, which is used as the timestamp of the data; the processing method of the alignment unit specifically includes: obtaining the data quantity in the alignment buffer area; according to the data quantity and the timestamp condition, different processing paths are selected; when the alignment buffer area is empty or the data quantity is less than the preset threshold and there is no alignment unit with a timestamp of zero, the data in the data cache queue is aligned according to the data and the timestamp; when the alignment buffer area is empty, directly jump to the final processing step; when the data quantity is less than the preset threshold and there is an alignment unit with a timestamp of zero, the alignment unit is emptied; when the data quantity is greater than or equal to the preset threshold, the data in the data cache queue is split into single alignment units and processed in the alignment order; the processed alignment units are written into the alignment buffer area in the timestamp order.

2. A multi-source heterogeneous bus data parallel acquisition recording system for implementing the method of claim 1, characterized in that, It includes: The acquisition and storage general control computer is used for unified configuration and management of heterogeneous bus data acquisition tasks; A plurality of heterogeneous bus interface acquisition modules, each module corresponding to processing one or more types of bus data; The acquisition and storage general control computer is connected with each heterogeneous bus interface acquisition module through POE high-speed Ethernet, realizing software-defined configuration of acquisition module parameters, acquisition timing control and transmission and collection of acquisition data; wherein, the general control computer runs the multi-source heterogeneous bus data parallel acquisition recording method, and each heterogeneous bus interface acquisition module executes the corresponding data acquisition task; the data normalization processing module is used for converting the heterogeneous data collected from different bus types into a unified data format; the data normalization processing module adds a unified packet header structure to each acquisition data packet, including bus type identification, data source identification, timestamp information, data length and verification information; wherein, the acquisition data packet in the unified data format is transmitted and stored in the system through a standardized transmission protocol, which is used for subsequent data processing and analysis; the large-capacity data storage module is used for storing the processed acquisition data packet in a high-speed streaming mode; the data index management module is used for establishing a multi-dimensional index based on the timestamp and the bus type, supporting efficient data retrieval after the test; wherein, the system supports correlation analysis and reconstruction of multi-source heterogeneous bus data after the test according to the index, realizing post-processing analysis of test data.

Images