Method, device, and program product for multi-source streaming data parsing
By using a unified data processing module with multi-threaded design and memory mapping technology, the problems of protocol heterogeneity and data processing inconsistency in traditional aircraft flight test systems are solved. This enables efficient and reliable data parsing and expansion capabilities, reduces system maintenance costs, and improves data processing performance and real-time performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-23
- Publication Date
- 2026-06-23
AI Technical Summary
Traditional aircraft flight test data processing systems suffer from protocol heterogeneity and system rigidity, unreliable real-time streaming data processing, low efficiency in parsing large offline files, and a disconnect between online and offline systems. These issues result in poor system scalability, high maintenance costs, and difficulty in meeting real-time requirements.
Employing multi-threaded design and memory mapping technology, it processes real-time network data streams and offline data files through a unified frame recognition and parsing module, supports plug-and-play protocol extensions, and monitors data link status in real time to ensure system stability.
It achieves technical unification of online and offline data processing, significantly improves data processing performance and system reliability, reduces maintenance costs, and ensures the real-time performance and data analysis efficiency of the system under high load.
Smart Images
Figure CN122268967A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer science and technology, and in particular to a method, apparatus, computer-readable storage medium, and program product for multi-source streaming data parsing. Background Technology
[0002] Traditional aircraft flight test data processing systems typically employ the following two separate architectures: Real-time data acquisition: For real-time data streams, a parsing module is independently developed and coded for each sensor protocol, receiving network or serial port data through single-threaded or simple multi-threaded methods. The system typically lacks intelligent protocol recognition capabilities; adding sensors requires modifying the core code and redeploying.
[0003] Offline parsing: For massive data files stored in the database, a separate, custom-developed parsing tool is used. These tools typically read data sequentially based on traditional file I / O interfaces. The offline parsing logic may differ from the real-time acquisition and parsing logic, and it is inefficient and time-consuming when processing files of several gigabytes or even tens of gigabytes.
[0004] However, traditional aircraft flight test data processing systems have the following drawbacks: Protocol heterogeneity and system rigidity: Faced with dozens of different devices and communication protocols of various formats, such as inertial navigation, MEMS, atmospheric data machines, radio altimeters, engine numerical control systems, generators, and landing gear controllers, existing systems require "tailor-made" parsing code for each protocol. This results in high system coupling, poor scalability, and an inability to adapt to the rapid iteration of new models.
[0005] Real-time streaming data processing is unreliable: it has weak capabilities in handling "packet merging" and "partial packet" issues in network byte streams and lacks robust frame synchronization mechanisms. The single-threaded architecture cannot cope with the concurrent pressure of multiple high-speed data streams, which can easily lead to data congestion and loss, and cannot meet the millisecond-level real-time requirements of flight testing.
[0006] Offline large file parsing is inefficient: using traditional sequential I / O methods to process extremely large files results in frequent disk accesses, data copying, and context switching overhead. Parsing a file of a GB in size often takes several hours, severely limiting the efficiency of data analysis.
[0007] The online and offline systems are disconnected: real-time data acquisition and post-analysis are implemented using two separate software systems. Inconsistencies in protocol parsing logic and data format output increase system complexity, maintenance costs, and the risk of errors. Protocol extensions require modifications to both systems, leading to repetitive work and potential for deviations.
[0008] The system suffers from low maintainability and automation: protocol extensions and network parameter configurations typically require source code modification and recompilation, making "hot-swappable" or configurable deployment impossible. Furthermore, the system lacks self-monitoring and self-recovery capabilities, necessitating significant time investment by technical personnel for operation and maintenance. Summary of the Invention
[0009] Based on the above analysis, the present invention aims to provide a method, apparatus, computer-readable storage medium, and program product for multi-source streaming data parsing, in order to solve the problems of fragmented real-time and offline processing, low efficiency in parsing large files, and unreliable processing in high-concurrency scenarios in existing aviation test data processing systems.
[0010] In a first aspect, embodiments of the present invention provide a method for parsing multi-source streaming data, the method comprising the following steps: Receives data including real-time network data streams and offline data files; The data is read and frame identification is performed uniformly based on the frame header feature code, length field and check code in the predefined frame structure, and complete data packets are cut out from the data; Extract the message ID from the data packet, and look up the data object and business object bound to the message ID in the parsing mapping table; the data object includes computer-readable structured fields, and the business object includes engineering values with physical meaning; call the templated parsing function to parse the data packet to the data object; call the static adaptation function corresponding to the message ID to parse the data object to the business object, and generate a parsing result, which includes the business object.
[0011] Based on the above solution, the present invention also makes the following improvements: Furthermore, the method also includes: Define and add the message ID, data object, and business object corresponding to the protocol, and add a corresponding static adapter function in the protocol adapter; Register the message ID corresponding to the added protocol in the parsing mapping table, and call the templated parsing function and the static adaptation function corresponding to the message ID to complete the parsing of the added protocol.
[0012] Furthermore, the method also includes: The received real-time network data stream is cached in the first thread-safe queue; Store the parsing results into the second thread-safe queue; The parsing result is obtained from the second thread-safe queue, and the parsing result is displayed graphically in real time by the display thread, and the parsing result is stored in the form of a binary file by the storage thread.
[0013] Furthermore, the method also includes: Memory mapping technology is used to map offline data files, either as a whole or in blocks, into the virtual address space of a process; file data is then accessed directly by offset within the virtual address space. Store the parsing result into the second thread-safe queue; The parsing result is obtained from the second thread-safe queue and converted into a structured file output by the protocol parser.
[0014] Furthermore, the method also includes network self-monitoring and recovery of the real-time network data stream, including: The connection status of the real-time network data stream is monitored in real time, and a reconnection mechanism or alarm information is automatically triggered when a network interruption or data reception timeout is detected.
[0015] Furthermore, the method also includes the physically meaningful engineering values as quantitative parameters characterizing the operating status of equipment in the field of aerospace testing, including at least one of the following: Acceleration, velocity, height, angle, pressure, temperature, rotational speed.
[0016] Secondly, embodiments of the present invention provide an apparatus for multi-source streaming data parsing, the apparatus comprising: The data input module is used to receive data including real-time network data streams and offline data files; The unified adaptive parsing module includes: a frame recognition module and a route parsing module; The frame recognition module is used to read the data and perform unified frame recognition based on the frame header feature code, length field and check code in the predefined frame structure, and cut out the complete data packet from the data; The routing parsing module is used to extract the message ID from the data packet, look up the data object and business object bound to the message ID in the parsing mapping table; the data object includes computer-readable structured fields, and the business object includes engineering values with physical meaning; call a templated parsing function to parse the data packet to the data object; call a static adaptation function corresponding to the message ID to parse the data object to the business object, and generate a parsing result, the parsing result including the business object.
[0017] Thirdly, embodiments of the present invention provide a device for multi-source streaming data parsing, comprising: The invention includes at least one processor, at least one memory, and a communication component; the memory stores a computer program that, when executed by the processor, implements the method described in at least one of the first aspects of the invention.
[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium for multi-source streaming data parsing, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed, it implements the method described in at least one of the first aspects of the present invention.
[0019] Fifthly, embodiments of the present invention provide a program product for multi-source streaming data parsing, the program product including a stored computer program, wherein when the computer program is run by a processor, the computer program code instructions cause the program product to perform the method described in at least one of the first aspects of the present invention.
[0020] The beneficial effects of this technical solution are: 1. Achieved technical unification for online and offline data processing. By using a unified parsing module, the technical gap between real-time systems and post-event analysis tools has been completely eliminated. A single codebase and a single protocol description serve both scenarios simultaneously, significantly reducing the overall development and maintenance costs of the system (estimated to be over 50%), while ensuring consistency throughout the entire data processing workflow.
[0021] 2. Possesses strong protocol adaptability and scalability. It supports plug-and-play functionality for over 20 aviation sensor protocols, reducing protocol extension time from the traditional 3-5 days to 2-4 hours. It only requires plug-in development and does not require modification of core framework code such as data acquisition, network communication, file I / O, and queue management, achieving highly efficient extension. The protocol extensions also apply to data containing both real-time network data streams and offline data files, ensuring consistent results for parsing both types of data using the same protocol.
[0022] 3. Significantly improves data processing performance The system employs a multi-threaded design, separating the data receiving, parsing, storage, and display threads to achieve asynchronous processing and ensure the system's real-time performance and responsiveness under high load. By adopting memory mapping technology, the time to parse GB-level offline historical data files has been reduced from several hours to tens of minutes, improving data analysis efficiency several times over.
[0023] 4. The system has high reliability and good maintainability. Real-time monitoring of the status of each data link. Automatic reconnection or alarm triggering upon detection of network interruption or data timeout ensures long-term stable system operation.
[0024] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description
[0025] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0026] Figure 1 A flowchart of the data parsing method provided in the embodiments of the present invention; Figure 2 This is a structural block diagram of the data parsing device provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of another data parsing device provided in an embodiment of the present invention. Detailed Implementation
[0027] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0028] Example 1 This embodiment discloses a data parsing method, the process of which is as follows: Figure 1 As shown, it includes the following steps: Step 101: Receive data including real-time network data streams and offline data files.
[0029] In the specific implementation process, two types of data are received from the original data, and the unified adaptive parsing module reads the data, which includes real-time network data streams and offline data files.
[0030] The real-time network data stream receives binary byte streams from various aviation sensors in real time through communication interfaces such as TCP / IP or serial ports.
[0031] Examples include various types of aviation sensors such as inertial navigation systems, MEMS, atmospheric data computers, and radio altimeters.
[0032] The offline data files are pre-stored on the disk and contain binary files of historical flight test data.
[0033] For example, the binary file is a file in the .dat format.
[0034] It should be noted that a thread-safe queue is used for receiving real-time network data streams, and the received real-time network data streams are cached in the first thread-safe queue. It should be noted that, in order to support efficient access to large files, memory mapping technology is used to receive offline data files, mapping the entire offline data file or its blocks to the virtual address space of the process; the file data is accessed directly by offset in the virtual address space.
[0035] Specifically, by calling the mmap() function in Linux or the CreateFileMapping() function in Windows, offline data files of several gigabytes or even larger sizes can be directly mapped into the virtual address space of the current process. After mapping, the file content can be directly accessed like a regular memory array. Data is then read using the pointer uchar* pBuf offset, completely avoiding the data copying issues between kernel mode and user mode and the context switching overhead caused by the traditional read() function.
[0036] Specifically for extremely large files, the system employs a block mapping and sliding processing strategy, mapping a fixed-size data block (e.g., 256MB) at a time, unmapping it after processing, and then mapping the next block. This keeps memory consumption constant and extremely low (only a few MB to tens of MB), thereby reducing the time to parse GB-level files from several hours in the traditional method to tens of minutes, improving performance by an order of magnitude.
[0037] Step 102: Read the data and perform unified frame identification based on the frame header feature code, length field and check code in the predefined frame structure, and cut out the complete data packet from the data.
[0038] In practical implementation, the unified adaptive parsing module is implemented by the Parser class and its core function ParserFromFileBuf(). This core function reads both real-time network data streams and offline data files simultaneously. Specifically, it reads the real-time network data stream from the first thread-safe queue and the offline data file from the virtual address space, extracting data packets from both types of data. Specifically, this core function includes a unified feature frame recognition function isHead(). By calling this function and based on the frame header feature code, length field, and checksum in the predefined frame structure, it segments complete data packets from the data. Unified frame recognition fundamentally solves the common "packet merging" and "incomplete packet" problems in real-time network data streams. "Packet merging" refers to multiple packets being stuck together, while "incomplete packet" refers to a packet being truncated.
[0039] It should be noted that the predefined general frame structure is a universal binary frame structure, which includes at least the following fields: [Frame Header Feature], [Length Field], [Message ID], [Timestamp], [Data Object], and [Check Field]. The frame header feature is used to quickly locate the start of a frame in the byte stream; the length field indicates the length of the entire data packet, used to completely segment the data packet from the stream; the message ID uniquely identifies a protocol or data type; the timestamp records the data generation time; the data object carries the specific protocol binary data; and the check field is used for CRC verification of data integrity. CRC stands for Cyclic Redundancy Check. Preferably, CRC16 is used for verification.
[0040] Specifically, firstly, frame start positioning is performed by scanning the data in the data using a unified feature frame recognition function to find the byte sequence that matches the frame header feature code in the predefined frame structure.
[0041] For example, a byte sequence that matches the frame header feature code in a predefined frame structure, such as the first byte being 0xAA / 0xAB / 0xAC, indicates the flight control channel, and the second byte being 0x55 / 0x56, indicates the C / M channel.
[0042] Secondly, the length is read. Once the frame header feature code is located, the length field is immediately read from a fixed offset position after the frame header.
[0043] For example, the length field could be an unsigned short length.
[0044] Finally, based on the length value read from the length field, data of the corresponding length is extracted from the beginning of the frame header and then verified by CRC to form a complete data packet.
[0045] Step 103: Extract the message ID from the data packet; look up the data object and business object bound to the message ID in the parsing mapping table; the data object includes computer-readable structured fields, and the business object includes engineering values with physical meaning; call the templated parsing function to parse the data packet to the data object; call the static adaptation function corresponding to the message ID to parse the data object to the business object, and generate a parsing result, which includes the business object.
[0046] In the specific implementation process, the system maintains a parsing mapping table (map structure) indexed by message ID. This parsing mapping table is pre-built by the Parser::InitParser() function during system initialization. It establishes the binding relationship between message ID and data object and business object. One message ID corresponds to one data object, and one data object corresponds to one business object.
[0047] For example, the extracted message ID is such as unsigned short msgID, the parsed mapping table is such as mCsvDirector, the templated parsing function is such as ParseBuffer(), and the static adaptation function is such as ParseData().
[0048] Furthermore, after extracting the message ID from the segmented complete data packet, the data object and business object bound to the message ID are searched in the parsing mapping table using the message ID as an index; the templated parsing function is called to parse the data packet into the data object, that is, according to the protocol format, the byte stream of the data packet is decomposed into the structured fields of the data object that the program can understand.
[0049] Furthermore, the static adaptation function corresponding to the message ID is invoked to parse the data object into the business object. This converts the binary data in the structured fields of the data object into meaningful engineering values for that business object within the business domain. Specifically, it performs unit conversions and status parsing on the binary data in the data object, transforming it into engineering values for a business object with actual physical meaning. Finally, the parsing of the data packet is completed through a templated parsing function and the static adaptation function corresponding to the message ID.
[0050] For example, for inertial navigation data, MEMS data, atmospheric data, radio altimeter data, etc. collected by sensors, taking the collected inertial navigation data as an example, a message ID is defined for the inertial navigation data. The data object corresponding to the message ID represents the raw byte array, status bit, unprocessed integer or floating-point value corresponding to the inertial navigation data. The business object corresponding to the data object represents the engineering values with physical units in the inertial navigation data, such as acceleration, velocity, angle, and calculated status.
[0051] It should be noted that this invention supports plug-in extensions. When adding a protocol, developers define the message ID, data object, and business object corresponding to the added protocol, and add a corresponding static adapter function in the protocol adapter. The message ID corresponding to the added protocol is registered in the parsing mapping table, and the templated parsing function and the static adapter function corresponding to the message ID are called to complete the parsing of the added protocol. This plug-in extension supports plug-and-play functionality for more than 20 aviation sensor protocols, reducing the protocol extension time from the traditional 3-5 days to 2-4 hours. Only plug-in development is required, without modifying the core framework code such as data acquisition, network communication, file I / O, and queue management, achieving high-efficiency extension. Furthermore, the protocol extension simultaneously applies to data containing both real-time network data streams and offline data files, ensuring consistent parsing results for both types of data using the same protocol.
[0052] For example, the following section details the "plug-and-play" extension process by adding a "Landing Gear Controller Data Voting Value (RetLGC)" protocol to the system: (1) Define and add the message ID and data object of the protocol. In the RECV_CMD namespace of the datadef.h file, define the message ID for the new protocol: const unsignedshort Ret_LGC = 0xXXXX; (e.g., 0x0310).
[0053] Create a header file retlgc.h, and define the binary data object class RetLGC (inherited from Request) and the business object class RetLGCParser (inherited from RequestParser).
[0054] / / retlgc.h class RetLGC : public Request { public: int SerializerBuffer(char* pBuf, int len); / / Serialization int LoadBuffer(char* pBuf, int len); / / Deserialization (called by the system kernel) / / ... Protocol-specific fields, such as data_1[3], data_2[4]... }; class RetLGCParser : public RequestParser { public: virtual void WriteCsvHead(QFile* pFile); / / Write CSV header virtual void WriteCsvItem(QFile* pFile); / / Write CSV data rows / / ... Business fields, such as LBP, RBP, LWS, RWS... }; The above function is implemented using LoadBuffer in retlgc.cpp, specifically describing how to parse the binary stream into the various fields of the binary data object class RetLGC.
[0055] (2) Define the business object and add a corresponding static adapter function in the protocol adapter. In classadapter.h and classadapter.cpp, add a new static adapter function to convert the data of the data object RetLGC to the fields of the business object RetLGCParser.
[0056] / / classadapter.h static bool ParseData(RetLGC* pData, RetLGCParser* ret); / / classadapter.cpp bool ClassAdapter::ParseData(RetLGC* pData, RetLGCParser* ret) { / / Copy public header information CopyHead(pData, ret); / / Perform business transformations, such as converting raw integer data into engineering values (e.g., acceleration, speed, height, angle, pressure, temperature, rotational speed). ret->LBP = (pData->data_2[0]&0xFFF) * 0.01; / / Example conversion / / ... other field conversions return true; } (3) Register in the unified adaptive parsing module In the switch(msgID) structure of the unified adaptive parsing module Parser::ParserFromFileBuf (which serves to parse both real-time network data streams and offline data files), add a processing branch for the new msgID (RECV_CMD::Ret_LGC).
[0057] / / parser.cpp (within the Parser::ParserFromFileBuf function) case RECV_CMD::Ret_LGC: RetLGC* pRetLGC = new RetLGC(); / / Create a binary data object ParseBuffer(pBuf, len, pRetLGC); / / Calls the same templated parsing function for both network data streams and offline data files. pRetLGCParser->AddItem(pRetLGC); / / Add to the list, calling the static adapter function added in the adapter. break It should be noted that a thread-safe queue is used for parsing and outputting real-time network data streams. The parsing results are stored in a second thread-safe queue. The parsing results are retrieved from the second thread-safe queue, and the parsing results are displayed graphically in real time by a display thread. The parsing results are stored in binary file format by a storage thread.
[0058] It should be noted that for parsing and outputting offline data files, the parsing results are stored in a second thread-safe queue; the parsing results are retrieved from the second thread-safe queue and converted into structured file output by the protocol parser.
[0059] Step 104: Perform network self-monitoring and recovery on the real-time network data stream.
[0060] In practice, the connection status of the real-time network data stream is monitored in real time. When a network interruption or data reception timeout is detected, a reconnection mechanism is automatically triggered or an alarm message is generated.
[0061] Example 2 This embodiment discloses a device for multi-source streaming data parsing, the structural block diagram of which is shown below. Figure 2 As shown, it includes: Data input module 201 and unified adaptive parsing module 203; wherein, The data input module is used to receive data including real-time network data streams and offline data files; The unified adaptive parsing module includes: The frame recognition module is used to read the data and perform unified frame recognition based on the frame header feature code, length field and check code in the predefined frame structure, and cut out the complete data packet from the data; The routing parsing module is used to extract the message ID from the data packet, look up the data object and business object bound to the message ID in the parsing mapping table, wherein the data object includes computer-readable structured fields and the business object includes engineering values with physical meaning; call the templated parsing function to parse the data packet to the data object; call the static adaptation function corresponding to the message ID to parse the data object to the business object, and generate a parsing result, wherein the parsing result includes the business object.
[0062] Furthermore, the device also includes a protocol plugin library 204 and a data processing output layer 205; wherein, The protocol plugin library, connected to the unified adaptive parsing module, is used to define the message ID, data object, and business object corresponding to the added protocol, and to add the corresponding static adaptation function in the protocol adapter; register the message ID corresponding to the added protocol in the parsing mapping table, and call the templated parsing function and the static adaptation function corresponding to the message ID; The data processing output module includes: A thread-safe queue is used to cache the parsing results of the unified adaptive parsing module; The display thread module is used to obtain the parsing results of the real-time network data stream from the thread-safe queue and display them graphically in real time. A storage thread module is used to obtain the parsing results of the real-time network data stream from the thread-safe queue and store them in the form of a binary file; A protocol parser is used to obtain the parsing results of the offline data file from the second thread-safe queue and convert them into structured file output.
[0063] Furthermore, the device also includes a system support module 202, which includes: The network monitoring module is used to monitor the connection status of the real-time network data stream in the data. The configuration management module is used to perform reconnection and start / stop data link services when a network interruption or data reception timeout is detected. The service management module is used to generate alarm information.
[0064] Example 3 This embodiment discloses another apparatus for multi-source streaming data parsing, the structural block diagram of which is shown below. Figure 3 As shown, it includes: The system comprises a data input layer 301, a unified adaptive parsing layer 302, a protocol plugin library 303, a data processing output layer 304, and a system support layer 305; among which, The data input layer is used to receive data, including real-time network data streams and offline data files. The unified adaptive parsing layer includes functions such as feature code recognition, data serialization, data deserialization, and data verification algorithms, which are used to parse the input data; The protocol plugin library includes protocol adapters, request data sets, and parsing request sets, which are used to standardize the data to be parsed; The data processing output layer includes a thread-safe queue, a display thread module, a storage thread module, and a protocol parser, which are used for efficient data output and distribution. The system support layer includes network monitoring, configuration management, and service management, which are used to monitor the connection status of the real-time network data stream in real time. When a network interruption or data reception timeout is detected, a reconnection mechanism is automatically triggered or an alarm message is generated.
[0065] Example 4 This embodiment discloses a device for multi-source streaming data parsing, including at least one processor, at least one memory, and a communication component; the memory stores a computer program, which, when executed by the processor, implements the method described in at least one of the specific embodiments of the present invention.
[0066] Example 5 This embodiment discloses a computer-readable storage medium for multi-source streaming data parsing, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed, it implements the method described in at least one of the specific embodiments of the present invention.
[0067] Example 6 This embodiment discloses a program product for multi-source streaming data parsing, including program code instructions. When the program product is executed by a computing device, the program code instructions cause the computing device to execute at least one of the methods described in Specific Embodiment 1 of the present invention.
[0068] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for parsing multi-source streaming data, characterized in that, The method includes the following steps: Receives data including real-time network data streams and offline data files; The data is read and frame identification is performed uniformly based on the frame header feature code, length field and check code in the predefined frame structure, and complete data packets are cut out from the data; Extract the message ID from the data packet, look up the data object and business object bound to the message ID in the parsing mapping table, the data object includes computer-readable structured fields, and the business object includes engineering values with physical meaning; call the templated parsing function to parse the data packet to the data object; call the static adaptation function corresponding to the message ID to parse the data object to the business object, and generate a parsing result, the parsing result including the business object.
2. The method according to claim 1, characterized in that, The method further includes: Define and add the message ID, data object, and business object corresponding to the protocol, and add a corresponding static adapter function in the protocol adapter; Register the message ID corresponding to the added protocol in the parsing mapping table, and call the templated parsing function and the static adaptation function corresponding to the message ID to complete the parsing of the added protocol.
3. The method according to claim 1, characterized in that, The method further includes: The received real-time network data stream is cached in the first thread-safe queue; Store the parsing results into the second thread-safe queue; The parsing result is obtained from the second thread-safe queue, and the parsing result is displayed graphically in real time by the display thread, and the parsing result is stored in the form of a binary file by the storage thread.
4. The method according to claim 1, characterized in that, The method further includes: Memory mapping technology is used to map offline data files, either as a whole or in blocks, into the virtual address space of a process; file data is then accessed directly by offset within the virtual address space. Store the parsing result into the second thread-safe queue; The parsing result is obtained from the second thread-safe queue and converted into a structured file output by the protocol parser.
5. The method according to claim 1, characterized in that, The method further includes network self-monitoring and recovery of the real-time network data stream, including: The connection status of the real-time network data stream is monitored in real time, and a reconnection mechanism or alarm information is automatically triggered when a network interruption or data reception timeout is detected.
6. The method according to any one of claims 1 to 5, characterized in that, The physically meaningful engineering values are quantitative parameters characterizing the operating status of equipment in the field of aerospace testing, including at least one of the following: Acceleration, velocity, height, angle, pressure, temperature, rotational speed.
7. An apparatus for multi-source streaming data parsing, characterized in that, The device includes: The data input module is used to receive data including real-time network data streams and offline data files; The unified adaptive parsing module includes: a frame recognition module and a route parsing module; The frame recognition module is used to read the data and perform unified frame recognition based on the frame header feature code, length field and check code in the predefined frame structure, and cut out the complete data packet from the data; The routing parsing module is used to extract the message ID from the data packet, look up the data object and business object bound to the message ID in the parsing mapping table, wherein the data object includes computer-readable structured fields and the business object includes engineering values with physical meaning; call a templated parsing function to parse the data packet to the data object; call a static adaptation function corresponding to the message ID to parse the data object to the business object, and generate a parsing result, wherein the parsing result includes the business object.
8. A device for multi-source streaming data parsing, characterized in that, include: The system includes at least one processor, at least one memory, and a communication component; the memory stores a computer program that, when executed by the processor, implements the method for multi-source streaming data parsing as described in any one of claims 1 to 5.
9. A computer-readable storage medium storing a computer program, which, when executed, implements the method for multi-source streaming data parsing according to any one of claims 1-5.
10. A computer program product, characterized in that, The computer program product includes a stored computer program that, when executed by a processor, implements the method for multi-source streaming data parsing as described in any one of claims 1-5.