A data stream parsing method, system, device and storage medium

Abstract

The application discloses a data stream analysis method, system, device and storage medium, and relates to the technical field of industrial internet, and the method comprises the following steps: in response to an analysis instruction of a target record data stream, acquiring original data corresponding to the target record data stream; based on a preset data frame synchronization identifier, performing frame boundary positioning on the original data stream, and splitting the original data stream into a plurality of continuous data blocks comprising complete data frames according to the positioned frame boundary; calling a parameter decoding rule matched with the original data stream, performing grouped parallel analysis on the plurality of continuous data blocks, and generating a time sequence analysis result; through accurate frame positioning and splitting based on the data frame synchronization identifier, invalid data interference is avoided, and grouped parallel analysis is performed according to a parameter acquisition frequency, so that the analysis speed and system processing capacity are improved, and a regular and unified data basis is provided for subsequent data storage, query and analysis.

Description

Technical Field

[0001] This invention relates to the field of industrial internet technology, specifically to a data stream parsing method, system, device, and storage medium. Background Technology

[0002] Currently, for time-series data streams with frame structures (such as QAR data from aircraft fast access recorders and sensor data streams from industrial equipment), conventional parsing methods mostly employ a serial processing mode. Specifically, the system parses the parameters in the data stream sequentially according to predefined parsing rules and stores the full parsing results.

[0003] However, the aforementioned serial processing method suffers from low parsing efficiency. Due to the large number of parameters, high sampling frequency, and large data volume in the data stream, parsing one by one is time-consuming, and processing speed has become a bottleneck for dealing with massive data scenarios. Summary of the Invention

[0004] The main objective of this invention is to provide a data stream parsing method, system, device, and storage medium. By achieving precise frame location and segmentation based on data frame synchronization identifiers, interference from invalid data is avoided. Parallel parsing is performed in groups according to parameter acquisition frequency, improving parsing speed and system processing capacity. Simultaneously, standardized time-series parsing results containing recorded parameters, acquisition time, and physical values ​​are generated, providing a regular and unified data foundation for subsequent data storage, querying, and analysis.

[0005] To achieve the above objectives, the embodiments of this application provide the following technical solutions: According to a first aspect of the embodiments of this application, a method for parsing a data stream is provided, comprising: In response to a parsing instruction for a target record data stream, the original data corresponding to the target record data stream is obtained; Based on a preset data frame synchronization identifier, the original data stream is frame boundary located, and the original data stream is split into several continuous data blocks including complete data frames according to the frame boundaries obtained by the location. The parameter decoding rules that match the original data stream are invoked to perform grouped parallel parsing on the several consecutive data blocks, generating time-series parsing results; the time-series parsing results include the recording parameters defined by the parameter decoding rules, the acquisition time corresponding to the recording parameters, and the physical values ​​of the parameters obtained by decoding.

[0006] In one possible implementation, the method further includes, before invoking the parameter decoding rule: Based on the device parameter definition information corresponding to the source of the target recorded data stream, a set of decoding rules is generated; the set of decoding rules includes multiple decoding entries, and each decoding entry includes the position information of the corresponding recorded parameter in the data frame, the acquisition frequency, and the numerical conversion rules. The decoding rule set is processed by version identification to obtain a decoding rule set with version identification; Based on the data type identifier carried by the target record data stream, the parameter decoding rule that matches the data type identifier is called from the decoding rule set with version identifier.

[0007] In one possible implementation, grouped parallel parsing is performed on the plurality of consecutive data blocks to generate time-series parsing results, including: Based on the acquisition frequency corresponding to each decoding entry in the parameter decoding rules, decoding entries with the same acquisition frequency are divided into the same parsing group to obtain several parsing groups. The parsing groups are respectively assigned to independent parallel processing units, so that each parsing group corresponds to one parallel processing unit; Each of the parallel processing units performs parallel decoding on the complete data frame in the continuous data block according to the decoding entries in the corresponding parsing group, and extracts and converts the physical values ​​of each recorded parameter at the corresponding acquisition time. The time series analysis results are generated based on each recorded parameter, the corresponding acquisition time, and the physical value of the parameter.

[0008] In one possible implementation, the step of locating frame boundaries of the original data stream based on a preset data frame synchronization identifier, and then splitting the original data stream into several consecutive data blocks, each containing a complete data frame, according to the located frame boundaries, includes: The candidate synchronization start position is determined in the original data stream based on the data frame synchronization identifier; Starting from the candidate synchronization start position, the data at multiple fixed positions in the original data stream are checked sequentially at preset intervals to determine whether the data frame synchronization identifier can be matched in all of them. When multiple subsequent fixed positions match the data frame synchronization identifier, the corresponding data interval is determined as a complete data frame. Multiple complete data frames are combined to obtain the continuous data block.

[0009] In one possible implementation, after obtaining the continuous data blocks, the method further includes: According to the matching rules of the data frame synchronization identifier, the data frames in the consecutive data blocks are verified; Data frames that fail verification are identified as abnormal data; The location information of the abnormal data in the original data stream is stored in the time series analysis result.

[0010] In one possible implementation, after generating the timing analysis result, the method further includes: For each recorded parameter in the time series analysis result, identify the time period in which the physical value of the recorded parameter remains continuously unchanged in the time dimension, and obtain at least one continuously unchanged time period; For each of the continuous and unchanging time periods, a corresponding compressed record is generated and stored. The compressed record includes the identifier of the record parameter, the start time of the continuous and unchanging time period, the length of the time period, and the corresponding physical value of the parameter.

[0011] In one possible implementation, it also includes: In response to a data read command, the target record parameters to be queried are obtained, and the compressed record corresponding to the target record parameters is read. Based on the start time, time period length, and physical parameter values ​​in the compressed record, the time-series analysis results of the target record parameters are recovered.

[0012] According to a second aspect of the embodiments of this application, a data stream parsing system is provided, the system comprising: The data acquisition module is used to acquire the original data corresponding to the target record data stream in response to the parsing instruction of the target record data stream; The splitting module is used to locate the frame boundaries of the original data stream based on a preset data frame synchronization identifier, and split the original data stream into several continuous data blocks including complete data frames according to the located frame boundaries. The parallel parsing module is used to call the parameter decoding rules that match the original data stream, perform grouped parallel parsing on the several consecutive data blocks, and generate time-series parsing results; the time-series parsing results include the recording parameters defined by the parameter decoding rules, the acquisition time corresponding to the recording parameters, and the physical values ​​of the parameters obtained by decoding.

[0013] According to a third aspect of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0014] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided having computer-readable instructions stored thereon, which can be executed by a processor to implement the method described in the first aspect above.

[0015] In summary, this application provides a data stream parsing method, system, device, and storage medium. By responding to a parsing instruction for a target recorded data stream, the original data corresponding to the target recorded data stream is obtained; the original data stream is frame boundary located based on a preset data frame synchronization identifier, and the original data stream is split into several continuous data blocks, each containing a complete data frame, according to the located frame boundaries; a parameter decoding rule matching the original data stream is invoked to perform grouped parallel parsing on the several continuous data blocks, generating a time-series parsing result; the time-series parsing result includes the recording parameters defined by the parameter decoding rule, the acquisition time corresponding to the recording parameters, and the physical values ​​of the parameters obtained through decoding. By achieving accurate frame location and splitting based on the data frame synchronization identifier, interference from invalid data is avoided; and grouped parallel parsing according to the parameter acquisition frequency improves parsing speed and system processing capacity. Simultaneously, a standardized time-series parsing result containing recording parameters, acquisition time, and physical values ​​is formed, providing a regular and unified data foundation for subsequent data storage, querying, and analysis. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0017] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0018] Figure 1 A flowchart illustrating a data stream parsing method provided in an embodiment of this application; Figure 2 A complete execution flowchart is provided for the embodiments of this application; Figure 3 A schematic diagram of a data stream parsing system provided in an embodiment of this application; Figure 4 This paper shows a structural diagram of an electronic device provided in an embodiment of this application; Figure 5 A diagram of a computer-readable storage medium provided in an embodiment of this application is shown.

[0019] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0022] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0023] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0025] To address the challenges of large data volumes, fixed frame structures, and stringent verification in airborne recording data streams, traditional parsing methods often employ serial processing. This results in low frame location accuracy, slow parameter parsing speed, and interference between parameters from different acquisition frequencies. Furthermore, the parsed data is not organized sequentially, making it difficult to use directly for subsequent storage and analysis. This application introduces data frame synchronization markers for precise frame boundary positioning, enabling rapid segmentation of valid and complete data frames from the original data stream. This reduces the impact of invalid data on subsequent processing stages, improving the reliability and efficiency of data segmentation.

[0026] Figure 1 This application illustrates a data stream parsing method provided by an embodiment, comprising: Step S101: In response to the parsing instruction of the target record data stream, obtain the original data corresponding to the target record data stream; Step S102: Based on the preset data frame synchronization identifier, perform frame boundary positioning on the original data stream, and split the original data stream into several continuous data blocks including complete data frames according to the positioned frame boundaries; Step S103: Invoke the parameter decoding rules that match the original data stream, perform grouped parallel parsing on the several consecutive data blocks, and generate time-series parsing results; the time-series parsing results include the recording parameters defined by the parameter decoding rules, the acquisition time corresponding to the recording parameters, and the physical values ​​of the parameters obtained by decoding.

[0027] By employing parameter decoding rules matched to the data stream source for grouped parallel parsing, parallel processing of parameters at different acquisition frequencies can be achieved, fully utilizing system processing resources and significantly improving the overall parsing speed. Simultaneously, it avoids mutual interference between data of different densities during parsing, ensuring the accuracy of parameter conversion. After parsing, standardized time-series parsing results containing recording parameters, acquisition time, and parameter physical values ​​are directly generated, presenting the data in a time-series format. This eliminates the need for additional data processing, supporting subsequent storage, querying, and business analysis, effectively simplifying the data processing workflow. The overall method optimizes the data stream processing chain while ensuring accurate frame positioning and correct parameter parsing. It is adaptable to the decoding rule requirements of multiple models and versions, and lays a stable data foundation for the compressed storage and efficient reading of subsequent large-capacity time-series data. It is particularly suitable for airborne recording data parsing scenarios with high data volume and high standardization requirements.

[0028] In one possible implementation, before invoking the parameter decoding rule, the method further includes: generating a decoding rule set based on device parameter definition information corresponding to the source of the target record data stream; the decoding rule set includes multiple decoding entries, each decoding entry including the position information of the corresponding record parameter in the data frame, the acquisition frequency, and the value conversion rule; performing version identification processing on the decoding rule set to obtain a decoding rule set with a version identifier; and, based on the data type identifier carried by the target record data stream, invoking the parameter decoding rule matching the data type identifier from the decoding rule set with the version identifier.

[0029] To adapt to the data parsing requirements of different device models and sub-models, this application pre-constructs a set of decoding rules corresponding to the device before performing parameter parsing. The set of decoding rules is generated based on the device parameter definition information from which the target recorded data stream originates. Each decoding entry in the set clearly records the position information of the corresponding recorded parameter in the data frame, the parameter acquisition frequency, and the value conversion rules. The generated set of decoding rules is version-identified, forming a set of decoding rules with version identifiers. This version differentiation enables effective management of different configuration parameter information for different device models, while retaining historical data traces of parameter information for easy tracking and maintenance. During parsing, based on the data type identifier carried by the target recorded data stream itself, the matching parameter decoding rules are called from the set of decoding rules with version identifiers. This achieves precise adaptation between the decoding rules and the data stream type, satisfying the need for parallel parsing of QAR data from multiple device models while avoiding parsing errors caused by rule mismatches, thus improving the universality and accuracy of the parsing process.

[0030] In one possible implementation, for example, the step of invoking a parameter decoding rule matching the data type identifier carried by the target record data stream specifically includes: extracting the data type identifier and the aircraft type identifier carried by the original data stream, wherein the data type identifier is used to distinguish different types of data streams, and the aircraft type identifier is used to distinguish different aircraft types and sub-aircraft types; selecting parameter decoding rules that match both the data type identifier and the aircraft type identifier from a set of decoding rules with version identifiers; validating the selected parameter decoding rules, and if the validation passes, determining to invoke the parameter decoding rule to perform subsequent grouped parallel parsing operations.

[0031] In one possible implementation, the version identification processing of the decoding rule set specifically includes: assigning a unique version number to the decoding rule set corresponding to each machine model and its sub-models, wherein the version number is associated with the machine model identifier of the target record data stream, so as to facilitate the quick retrieval of the corresponding version of the decoding rule set according to the machine model.

[0032] In one possible implementation, in step S101, the step of obtaining the original data corresponding to the target record data stream in response to the parsing instruction of the target record data stream specifically includes: receiving the target record data stream parsing instruction issued by the user, wherein the parsing instruction carries the identification information of the target record data stream, the data source identification, and the parsing range; locating the storage location of the target record data stream according to the identification information in the parsing instruction; and extracting the original data matching the parsing range from the storage location.

[0033] In one possible implementation, the parsing range in the parsing instruction includes the data acquisition time period and the target parameter type. When extracting raw data, only the raw data related to the target parameter type within the acquisition time period is extracted, and irrelevant data is removed to reduce the pressure of subsequent processing.

[0034] In one possible implementation, after obtaining the raw data, a raw data preprocessing step is further included: the extracted raw data is deduplicated and denoised to remove duplicate data segments and interference data, ensuring the integrity and accuracy of the raw data.

[0035] In one possible implementation, in step S102, the step of locating the frame boundaries of the original data stream based on a preset data frame synchronization identifier, and splitting the original data stream into several continuous data blocks including complete data frames according to the located frame boundaries, includes: determining a candidate synchronization start position in the original data stream according to the data frame synchronization identifier; taking the candidate synchronization start position as the starting point, sequentially checking the data at multiple subsequent fixed positions in the original data stream at preset intervals to determine whether they can all match the data frame synchronization identifier; when multiple subsequent fixed positions all match the data frame synchronization identifier, determining the currently corresponding data interval as a complete data frame; and combining multiple complete data frames to obtain the continuous data block.

[0036] By employing multi-level synchronization flag verification to locate data frame boundaries, the reliability of frame recognition is effectively improved, avoiding misjudgments caused by single-position matching. First, candidate synchronization start positions are initially determined in the original data stream based on preset data frame synchronization flags. Using this as a benchmark, data at multiple fixed positions is sequentially verified at preset intervals to determine if the data at each position matches the synchronization flag. Only when all verification positions successfully match is the corresponding data interval recognized as a valid and complete data frame, thus excluding invalid data segments that do not conform to the frame structure. The selected complete data frames are then systematically combined to form continuous data blocks, providing a well-organized and reliable data foundation for subsequent parallel parsing. This ensures that the subsequent parsing process only targets valid data, improving overall processing efficiency and result accuracy.

[0037] In one possible implementation, the process of locating the frame boundary of the original data stream based on the preset data frame synchronization identifier further includes a data cleaning step: based on the matching rules of the data frame synchronization identifier, invalid data in the original data stream is removed, wherein the invalid data is a data segment that cannot be matched with the data frame synchronization identifier.

[0038] In one possible implementation, the preset interval is a fixed byte interval, the subsequent multiple fixed positions can be specifically four consecutive check positions, and the data frame synchronization identifier is a preset four fixed values. Only when the data at the four consecutive check positions matches the four fixed values ​​one by one is the corresponding data interval determined as a complete data frame.

[0039] In one possible implementation, after obtaining the continuous data block, the method further includes: verifying the data frames in the continuous data block according to the matching rules of the data frame synchronization identifier; identifying the data frames that fail the verification as abnormal data; and storing the location information of the abnormal data in the original data stream into the timing parsing result.

[0040] After splitting the continuous data blocks, the synchronization and verification of each data frame within them further filters out abnormal data that does not meet the preset format requirements, ensuring the reliability of subsequent parsing results. Verifying each frame within a data block according to the matching rules of the data frame synchronization identifiers allows for timely identification of data content with transmission errors or incorrect formats, preventing abnormal data from participating in the parameter decoding process and causing distortion of the parsing results. Data frames that fail verification are marked as abnormal data, and their location information in the original data stream is recorded in the time-series parsing results. This provides direct evidence for subsequent data investigation, fault analysis, and quality traceability. By retaining the location information of abnormal data, the entire parsing process is traceable, and targeted processing of abnormal points can be performed in subsequent data application stages, improving the completeness and practicality of the overall data stream analysis.

[0041] In one possible implementation, after storing the location information of the abnormal data in the original data stream into the time-series parsing result, the method further includes: counting the number of abnormal data, the number of normal data, and the proportion of abnormal data and normal data in the total amount of data in the original data stream, and synchronously storing the statistical results into the time-series parsing result.

[0042] In one possible implementation, step S103 involves performing grouped parallel parsing on the plurality of consecutive data blocks to generate a timing parsing result, including: dividing decoding entries with the same acquisition frequency into the same parsing group according to the acquisition frequency corresponding to each decoding entry in the parameter decoding rules, thus obtaining a plurality of parsing groups; allocating the plurality of parsing groups to independent parallel processing units, so that each parsing group corresponds to one parallel processing unit; through each of the parallel processing units, performing parallel decoding on the complete data frame in the consecutive data block according to the decoding entries in the corresponding parsing group, extracting and converting the physical values ​​of each recorded parameter at the corresponding acquisition time; and generating the timing parsing result based on each recorded parameter, the corresponding acquisition time, and the physical values ​​of the parameters.

[0043] This implementation optimizes the parsing process to address the issue of varying parameter acquisition frequencies. Decoding entries are grouped by acquisition frequency, with entries at the same frequency grouped together. This allows for unified processing of parameters with similar characteristics, reducing interference between data from different frequencies. Assigning different parsing groups to independent parallel processing units fully utilizes system resources, enabling synchronous decoding of multiple parameter groups and significantly improving overall parsing speed. This is particularly suitable for scenarios involving large volumes of airborne recording data streams. Each parallel processing unit performs parallel decoding on the complete data frame according to its corresponding decoding entry, accurately extracting and converting the physical values ​​of each recording parameter at the corresponding acquisition time, ensuring the accuracy and consistency of parameter parsing. After parallel decoding, all recording parameters, corresponding acquisition times, and physical parameter values ​​are integrated to form a time-series-arranged parsing result. This makes the data structure more organized, facilitating subsequent storage, retrieval, and business analysis, while also improving the standardization and efficiency of the entire parsing process.

[0044] In one possible implementation, after dividing the decoding entries with the same acquisition frequency into the same parsing group, the method further includes: dividing the decoding entries in the same parsing group into several sub-parsing groups according to a fixed number of entries, and each parallel processing unit performs decoding operations sequentially according to the sub-parsing groups to reduce data storage pressure.

[0045] In one possible implementation, after generating the time series analysis result, the method further includes: performing an integrity check on the time series analysis result, the check content including the integrity of the recorded parameters, the continuity of the acquisition time, and the rationality of the physical values ​​of the parameters. If the check fails, the grouped parallel analysis step is re-executed.

[0046] During the parallel parsing process, each parallel processing unit is independent of the others and does not interfere with each other. After each parallel processing unit completes the decoding of its corresponding parsing group, it outputs the parameter parsing result of that group. The output results of all parallel processing units are summarized and combined with the acquisition time of each recorded parameter to generate a complete time-series parsing result.

[0047] In one possible implementation, after generating the time series analysis result, the method further includes: for each record parameter in the time series analysis result, identifying a time period in which the physical value of the record parameter remains continuously unchanged in the time dimension, to obtain at least one continuously unchanged time period; generating and storing a corresponding compressed record based on each continuously unchanged time period, wherein the compressed record includes the identifier of the record parameter, the start time of the continuously unchanged time period, the time period length, and the corresponding physical value of the parameter.

[0048] To address the characteristic of stable physical values ​​of parameters in time-series analysis results over extended periods, this implementation method performs targeted compression on the time-series data to reduce storage footprint and improve subsequent read / write efficiency. By iterating through each record parameter sequentially, time periods where the physical values ​​of the parameters remain continuously unchanged over time are identified. This fully leverages the redundant features of the time-series data, reducing the storage overhead of duplicate data. Based on the identified continuous unchanging time periods, corresponding compressed records are generated, retaining only the parameter identifier, start time, time period length, and corresponding physical value of the parameter. This streamlined information replaces a large number of repetitive sampling points in the original time-series data, significantly reducing the data volume without losing valid data content. Storing the compressed records uniformly effectively alleviates the storage pressure caused by large-capacity record data streams, while providing a lighter data carrier for subsequent rapid retrieval and reading, thus improving overall data management and usage efficiency.

[0049] In one possible implementation, storing the compressed record specifically involves storing the compressed record in a preset big data storage system. This big data storage system is adapted to the large-capacity storage requirements of QAR time-series data streams, thus avoiding the problem of insufficient storage capacity in traditional relational databases.

[0050] In one possible implementation, the method further includes: in response to a data reading instruction, obtaining the target record parameter to be queried, and reading the compressed record corresponding to the target record parameter; and recovering the time-series analysis result of the target record parameter based on the start time, time period length and parameter physical value in the compressed record.

[0051] This implementation provides a corresponding read and restore mechanism for compressed and stored record data, ensuring that the compressed data can be completely restored to the original time-series analysis results, achieving storage optimization without affecting normal data use. Upon receiving a data read command, the system first determines the parameters of the target record to be queried, and then retrieves the corresponding compressed record based on the parameter information, ensuring the targeted and efficient data read. Based on the start time, time period length, and parameter physical values ​​contained in the compressed record, the data is completely restored according to time-series rules, restoring the time-series changes of the target record parameters throughout the entire acquisition period. The entire restoration process does not rely on additional auxiliary information to accurately restore all valid data, enabling the lightweight compressed data to meet the normal needs of subsequent query, display, and analysis applications, balancing data storage efficiency and ease of use.

[0052] In one possible implementation, reading the compressed record corresponding to the target record parameter specifically involves: reading all compressed records corresponding to the target record parameter from the big data storage system, performing integrity verification on the read compressed records, and only executing subsequent recovery steps after the verification passes.

[0053] In one possible implementation, the frame boundary positioning, grouped parallel parsing, and time-series parsing result compression storage are performed in a pipelined continuous processing manner. The output of the parsing step is directly used as the input of the subsequent compression step, and the full intermediate parsing results are not stored, thereby further improving data processing efficiency.

[0054] The method provided in this application embodiment performs frame-level parsing of data based on the data frame synchronization word in the original data, and completes data compression processing during the parsing process. The overall process can be combined with the appendix. Figure 2 Please provide a detailed explanation. Figure 2 The complete execution flow is shown on one side, which is the decoder construction process, and on the other side, which is the main QAR data processing flow. The decoder is generated from the aircraft QAR decoding parameters and is further divided into acquisition rules and decoding rules to provide rule support for subsequent parsing. The main QAR data processing flow starts from the aircraft QAR data and goes through data cleaning and splitting, fast parsing, grouped parallel parsing, data frame abnormal data marking, data compression, and data storage in sequence to complete the entire processing flow.

[0055] The implementation of this method is divided into five stages, each with smooth transitions and clear logic. The following is a detailed explanation of each stage using an example of QAR data processing on the A320 model.

[0056] Phase 1: Building the decoder.

[0057] First, based on the parameter definition information provided by the aircraft manufacturer or equipment supplier, a decoder corresponding to the aircraft model is constructed. The decoder includes multiple decoder entries, each of which contains at least the parameter acquisition frequency, the byte position of the parameter in the data frame, the data type, and the data conversion rules. At the same time, the decoder is decoupled from the parsing program, and a versioned management method is used to support the parallel parsing of QAR data from multiple aircraft models, which facilitates the traceability and maintenance of parameter information.

[0058] Specifically for the A320 aircraft, based on the parameter definitions provided by Airbus, separate decoders were constructed for the CEO and NEO sub-models. The CEO decoder contains approximately 2,400 decoder entries, while the NEO decoder contains approximately 2,700. These decoder entries cover multiple frequency ranges, including 0.25Hz, 0.5Hz, 0.75Hz, 1Hz, 2Hz, 4Hz, 6Hz, 8Hz, 16Hz, and 64Hz. Byte positions range from 1 to 1024, and data types include linear, discrete, and ASCII types. Furthermore, the decoders are further divided into two branches: acquisition rules and decoding rules. Acquisition rules define the parameter sampling period and time base, while decoding rules define the data conversion logic and physical meaning. Both branches support the subsequent parsing process.

[0059] Phase Two: Data Cleaning and Frame-Level Splitting.

[0060] The core of this stage is to perform validity checks on the original data packets based on the data frame synchronization words in the original QAR data, remove invalid data that does not conform to the synchronization word rules, and then split the cleaned data stream into multiple data blocks according to the data frame boundaries. Each data block contains one or more complete consecutive data frames.

[0061] Specifically, for the raw binary data stream of the A320 aircraft's QAR (Quick Access Response) system, four preset data frame synchronization words are used for validity verification. The decimal values ​​corresponding to these four synchronization words are 585, 1464, 2631, and 3512, respectively, and their 16-bit binary representations are 0000001001000111, 0000010110111000, 0000101001000111, and 0000110110111000. During verification, the system traverses from the first byte of the raw data stream. After matching the first synchronization word, it sequentially verifies four consecutive check bits at fixed intervals. Only when all four positions match the corresponding synchronization word is the data segment considered a valid and complete data frame; otherwise, the invalid data segment is discarded. After completing the full data verification, the cleaned valid data stream is split according to the data frame boundaries, forming multiple continuous data blocks.

[0062] The third stage: parallel parsing based on decoding entry grouping.

[0063] This stage improves parsing efficiency by grouping decoding entries and employing a parallel parsing mechanism. The specific process is as follows: decoding entries are grouped according to their sampling frequency and data density, forming multiple decoding entry groups. For each decoding entry group, a parallel parsing mechanism is used to synchronously decode the data blocks, achieving parallel parsing of the data frame. Since different sampling frequencies correspond to different data densities—higher frequencies result in denser data acquisition, while lower frequencies result in sparser data—grouping by sampling frequency avoids interference between data of different densities during parsing.

[0064] In practice, decoding entries with the same acquisition frequency are grouped together; for example, parameters at 0.25Hz are grouped into one group, and parameters at 1Hz into another, thus forming multiple independent parsing groups. Each parsing group is then assigned to an independent parallel processing unit, employing a multi-threaded mechanism for synchronous decoding, fully utilizing the system's multi-core resources to improve parsing efficiency. Simultaneously, to reduce database write pressure, entries within each parsing group are divided into subgroups of 200 entries each, and decoding operations are performed sequentially by subgroup, ensuring both parsing speed and rapid data storage to the database. Each parallel processing unit decodes complete data frames in consecutive data blocks according to the corresponding decoding entries, extracting and converting the physical values ​​of each recorded parameter at the corresponding acquisition time. Finally, all parsing results are summarized to generate a time-series parsing result containing recorded parameters, acquisition time, and physical parameter values.

[0065] Phase 4: Abnormal data frame marking and processing.

[0066] Throughout the entire process of data splitting and parsing, each data frame is continuously verified twice according to the data frame synchronization word rules to ensure the integrity and reliability of the parsing results.

[0067] Specifically, since the read raw data is a byte array, with each byte consisting of 8 binary bits, the read binary data is filtered according to the synchronization words in the second stage: starting from the first byte, after matching the first synchronization word, it is checked whether the data composed of the 2048th and 2049th bytes satisfies the second synchronization word, and so on, until a data frame satisfying all four synchronization words is found; after finding a data frame, 8192 bytes are added to confirm the next frame, and so on to find all data frames satisfying the synchronization words. During the filtering process, data frames that do not meet the synchronization word rules are marked as abnormal frames, and their time position or frame sequence number information in the raw data stream is recorded. At the same time, the amount of normal data, the amount of abnormal data, and their proportion in the total data volume are counted and retained in the time series analysis results, providing a complete traceability basis for subsequent fault diagnosis and data quality analysis.

[0068] Fifth stage: Time-series compressed storage.

[0069] After parsing, the time series parsing results are organized into a multi-dimensional data structure of decoding entries, time series, and decoding values. For the same decoding entry, when its decoding value remains unchanged within a continuous time interval, only the start position, data volume, and corresponding decoding value of that time interval are recorded to form compressed time series data. The compressed data is then stored in the database. If necessary, the original parsing results can be restored using the corresponding decompression rules.

[0070] Because QAR data is massive time-series data, traditional relational databases such as MySQL, SQL Server, and Oracle can no longer meet the storage requirements. Therefore, big data storage solutions such as Click House are adopted to store the data. Taking into account the temporal continuity of QAR data, where data from some sensors remains unchanged over a period of time, data compression based on the collection frequency of each item can significantly reduce storage redundancy: during data storage, only the start time, time value, and data volume are stored; during data display, reverse expansion is performed to restore the complete time-series data. For example, for a parameter with a frequency of 1 Hz, if there are n consecutive identical values ​​of 'a' starting from position 'b', during storage, only the start position 'b', the value 'a', and the number 'n' are recorded. During display, the data is sorted in ascending order based on position, and each value is iterated and supplemented to restore the complete data.

[0071] In summary, this application uses the data frame synchronization word as the primary index for parsing, achieving frame-level accurate parsing of QAR data; it constructs a parallel parsing mechanism based on the decoding entry sampling frequency and data density grouping to improve parsing speed; during the parsing process, abnormal data frames are marked with a position-preserving method to ensure data traceability; and it constructs a sparse temporal compression model oriented towards flight parameter characteristics, reducing storage redundancy by recording continuous and unchanging time intervals; it pipelines the parsing and compression processes, eliminating the need to store all intermediate parsing results, further improving processing efficiency. Furthermore, frame boundary positioning, grouped parallel parsing, and temporal compression storage are executed continuously in a pipelined manner, with the output of the parsing step directly serving as the input of the compression step, eliminating the need to store all intermediate parsing results, further improving overall processing efficiency.

[0072] Compared with existing technologies, this method significantly reduces computational complexity through frame-level parallel parsing driven by synchronous words, controlling the parsing time of a single flight segment QAR data to the minute level, thus meeting the continuous parsing requirements of large-scale flight data. By preserving the location of abnormal frames and using time-series compressed storage, it ensures data integrity and traceability while effectively reducing storage pressure and avoiding the waste of resources in storing all intermediate data. It is fully adapted to the processing requirements of QAR data, improving the efficiency and practicality of data processing.

[0073] In summary, this application provides a data stream parsing method. By responding to a parsing instruction for a target recorded data stream, the method obtains the original data corresponding to the target recorded data stream; it locates the frame boundaries of the original data stream based on a preset data frame synchronization identifier, and then splits the original data stream into several continuous data blocks, each containing a complete data frame, according to the located frame boundaries; it calls a parameter decoding rule matching the original data stream to perform grouped parallel parsing on the several continuous data blocks, generating a time-series parsing result; the time-series parsing result includes the recording parameters defined by the parameter decoding rule, the acquisition time corresponding to the recording parameters, and the physical values ​​of the parameters obtained through decoding. By achieving accurate frame location and splitting based on the data frame synchronization identifier, interference from invalid data is avoided. Grouped parallel parsing according to the parameter acquisition frequency improves parsing speed and system processing capacity, while forming a standardized time-series parsing result containing recording parameters, acquisition time, and physical values, providing a regular and unified data foundation for subsequent data storage, querying, and analysis.

[0074] Based on the same technical concept, embodiments of this application also provide a data stream parsing system, such as... Figure 3 As shown, the system includes: The data acquisition module 301 is used to acquire the original data corresponding to the target record data stream in response to the parsing instruction of the target record data stream; The splitting module 302 is used to locate the frame boundaries of the original data stream based on a preset data frame synchronization identifier, and split the original data stream into several continuous data blocks including complete data frames according to the located frame boundaries. The parallel parsing module 303 is used to call the parameter decoding rules that match the original data stream, perform grouped parallel parsing on the several consecutive data blocks, and generate time-series parsing results; the time-series parsing results include the recording parameters defined by the parameter decoding rules, the acquisition time corresponding to the recording parameters, and the physical values ​​of the parameters obtained by decoding.

[0075] This application also provides an electronic device corresponding to the method provided in the foregoing embodiments. Please refer to... Figure 4 The diagram illustrates an electronic device provided by some embodiments of this application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202, and a communication interface 203, wherein the processor 200, the communication interface 203, and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can run on the processor 200, and when the processor 200 runs the computer program, it executes the method provided by any of the foregoing embodiments of this application.

[0076] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one physical port (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0077] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. After receiving an execution instruction, the processor 200 executes the program. The method disclosed in any of the foregoing embodiments of this application can be applied to the processor 200, or implemented by the processor 200.

[0078] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the above method.

[0079] The electronic devices and methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0080] This application also provides a computer-readable storage medium corresponding to the method provided in the foregoing embodiments. Please refer to... Figure 5The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored, which, when run by a processor, executes the methods provided in any of the foregoing embodiments.

[0081] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0082] The computer-readable storage medium provided in the above embodiments of this application and the method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application stored therein.

[0083] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for parsing a data stream, characterized in that, include: In response to a parsing instruction for a target record data stream, the original data corresponding to the target record data stream is obtained; Based on a preset data frame synchronization identifier, the original data stream is frame boundary located, and the original data stream is split into several continuous data blocks including complete data frames according to the frame boundaries obtained by the location. The parameter decoding rules that match the original data stream are invoked to perform grouped parallel parsing on the several consecutive data blocks, generating time-series parsing results; the time-series parsing results include the recording parameters defined by the parameter decoding rules, the acquisition time corresponding to the recording parameters, and the physical values ​​of the parameters obtained by decoding.

2. The method as described in claim 1, characterized in that, Before invoking the parameter decoding rule, the method further includes: Based on the device parameter definition information corresponding to the source of the target recorded data stream, a set of decoding rules is generated; the set of decoding rules includes multiple decoding entries, and each decoding entry includes the position information of the corresponding recorded parameter in the data frame, the acquisition frequency, and the numerical conversion rules. The decoding rule set is processed by version identification to obtain a decoding rule set with version identification; Based on the data type identifier carried by the target record data stream, the parameter decoding rule that matches the data type identifier is called from the decoding rule set with version identifier.

3. The method as described in claim 2, characterized in that, Perform grouped parallel parsing on the aforementioned consecutive data blocks to generate time-series parsing results, including: Based on the acquisition frequency corresponding to each decoding entry in the parameter decoding rules, decoding entries with the same acquisition frequency are divided into the same parsing group to obtain several parsing groups. The parsing groups are respectively assigned to independent parallel processing units, so that each parsing group corresponds to one parallel processing unit; Each of the parallel processing units performs parallel decoding on the complete data frame in the continuous data block according to the decoding entries in the corresponding parsing group, and extracts and converts the physical values ​​of each recorded parameter at the corresponding acquisition time. The time series analysis results are generated based on each recorded parameter, the corresponding acquisition time, and the physical value of the parameter.

4. The method as described in claim 1, characterized in that, The step of locating frame boundaries of the original data stream based on a preset data frame synchronization identifier, and then splitting the original data stream into several continuous data blocks, each containing a complete data frame, according to the located frame boundaries, includes: The candidate synchronization start position is determined in the original data stream based on the data frame synchronization identifier; Starting from the candidate synchronization start position, the data at multiple fixed positions in the original data stream are checked sequentially at preset intervals to determine whether the data frame synchronization identifier can be matched in all of them. When multiple subsequent fixed positions match the data frame synchronization identifier, the corresponding data interval is determined as a complete data frame. Multiple complete data frames are combined to obtain the continuous data block.

5. The method as described in claim 4, characterized in that, After obtaining the continuous data blocks, the process also includes: According to the matching rules of the data frame synchronization identifier, the data frames in the consecutive data blocks are verified; Data frames that fail verification are identified as abnormal data; The location information of the abnormal data in the original data stream is stored in the time series analysis result.

6. The method as described in claim 1, characterized in that, After generating the time series analysis results, the process also includes: For each recorded parameter in the time series analysis result, identify the time period in which the physical value of the recorded parameter remains continuously unchanged in the time dimension, and obtain at least one continuously unchanged time period; For each of the continuous and unchanging time periods, a corresponding compressed record is generated and stored. The compressed record includes the identifier of the record parameter, the start time of the continuous and unchanging time period, the length of the time period, and the corresponding physical value of the parameter.

7. The method as described in claim 6, characterized in that, Also includes: In response to a data read command, the target record parameters to be queried are obtained, and the compressed record corresponding to the target record parameters is read. Based on the start time, time period length, and physical parameter values ​​in the compressed record, the time-series analysis results of the target record parameters are recovered.

8. A data stream parsing system, characterized in that, The system includes: The data acquisition module is used to acquire the original data corresponding to the target record data stream in response to the parsing instruction of the target record data stream; The splitting module is used to locate the frame boundaries of the original data stream based on a preset data frame synchronization identifier, and split the original data stream into several continuous data blocks including complete data frames according to the located frame boundaries. The parallel parsing module is used to call the parameter decoding rules that match the original data stream, perform grouped parallel parsing on the several consecutive data blocks, and generate time-series parsing results; the time-series parsing results include the recording parameters defined by the parameter decoding rules, the acquisition time corresponding to the recording parameters, and the physical values ​​of the parameters obtained by decoding.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when running the computer program, performs an action to implement the method as claimed in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that can be executed by a processor to implement the method as described in any one of claims 1-7.

Images