A telemetry data stream parsing method and system based on a dynamic decoding rule library

Abstract

The application discloses a telemetry data stream analysis method and system based on a dynamic decoding rule library, relates to telemetry and data processing technology, and comprises the following steps: constructing a dynamic decoding rule library and a heuristic instruction set; sending corresponding heuristic instructions to a target data source according to a preset strategy or an adaptive sequence, and collecting telemetry data; receiving and buffering original byte streams from the corresponding data source; for any sent heuristic instruction, locating a candidate response data block in the original byte stream according to a sending time stamp and a preset response delay window; generating a corresponding candidate decoding rule for any candidate response data block based on the dynamic decoding rule library, and performing trial decoding by using the candidate decoding rule; and performing rationality evaluation to determine a decoding rule of the applicable telemetry data stream according to an evaluation result. The application provides an effective mechanism for actively, systematically guiding and automatically generating a trusted decoding rule.

Description

Technical Field

[0001] This application relates to the fields of telemetry and data processing technology, and in particular to a telemetry data stream parsing method and system based on a dynamic decoding rule base. Background Technology

[0002] Telemetry data stream parsing is a key technology in fields such as industrial equipment monitoring. Its basic principle is to convert received binary byte streams into parameters with actual physical meaning, such as temperature, voltage, and attitude, based on pre-defined, fixed decoding rules (usually called frame format definitions). Traditional parsing systems heavily rely on complete prior knowledge of the target device's communication protocol. Decoding rules typically exist in static configuration files or hard-coded form, and the system parses the data according to the synchronization header, field length, offset, data type, and calibration formula defined in the rules. This method is simple and effective in scenarios where the protocol is known and stable.

[0003] In many practical application scenarios, such as downlink data using private encryption, monitoring of non-cooperative targets, and integration with legacy systems, the telemetry data streams faced by the parser are often "black box" data with unknown protocols and encrypted or complexly encoded. Traditional static rule-based parsing methods are completely ineffective against such data streams. Therefore, some reverse engineering or blind parsing methods have emerged in existing technologies, attempting to infer the structure of the data stream from itself. Common methods rely on statistical analysis of static datasets, such as finding recurring fixed byte sequences as potential frame headers or tails, or using entropy analysis to distinguish data from padding fields.

[0004] Existing technologies are highly dependent on a large amount of sample data, making it difficult to carry out work when samples are scarce in the initial stage. Secondly, when the data stream is strongly encrypted or obfuscated, its statistical characteristics tend to be random noise, making the identification method based on statistical patterns extremely inefficient or even completely fail. Furthermore, existing methods can only make shallow inferences about the data format and cannot accurately map binary segments to specific physical quantities and their units of measurement, let alone verify the correctness of the mapping relationship. Summary of the Invention

[0005] This application provides a telemetry data stream parsing method and system based on a dynamic decoding rule base, proposing an effective mechanism for proactively and systematically guiding and automatically generating reliable decoding rules.

[0006] This application proposes a telemetry data stream parsing method based on a dynamic decoding rule base, including:

[0007] Construct a dynamic decoding rule base and a trial instruction set, wherein the dynamic decoding rule base initially contains general decoding meta-rules, and the trial instruction set contains instructions that produce observable physical effects on the state of the target data source;

[0008] According to a preset strategy or adaptive sequence, send corresponding probe commands to the target data source and collect telemetry data, recording the sending timestamp of each probe command.

[0009] Receive and buffer raw byte streams from the corresponding data source, and add an arrival timestamp to each raw byte stream;

[0010] For any sent probe command, the candidate response data block is located in the raw byte stream based on its sending timestamp and the preset response delay window;

[0011] Based on the dynamic decoding rule base, corresponding candidate decoding rules are generated for any candidate response data block, and trial decoding is performed using the candidate decoding rules;

[0012] The rationality of the physical quantity data sequence output from the trial decoding is evaluated, and the applicable decoding rules for the telemetry data stream are determined based on the evaluation results.

[0013] This application also proposes a telemetry data stream parsing system based on a dynamic decoding rule base, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the telemetry data stream parsing method based on the aforementioned dynamic decoding rule base.

[0014] This application employs a reverse parsing method of actively sending probing commands, observing responses, and performing correlation analysis. This method can purposefully induce regular changes in the data stream, thereby providing anchorable and causally related characteristic signals for parsing. It enables the systematic extraction of potential decoding rules from encrypted byte streams that are traditionally considered noise.

[0015] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0017] Figure 1 This is a schematic diagram of the basic process of the telemetry data stream parsing method based on the dynamic decoding rule base in this embodiment. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0019] This application proposes a telemetry data stream parsing method based on a dynamic decoding rule base, such as... Figure 1 As shown, it includes:

[0020] In step S101, a dynamic decoding rule base and a trial instruction set are constructed. The dynamic decoding rule base initially contains general decoding meta-rules, or it can be empty initially. The trial instruction set contains instructions that produce observable physical effects on the state of the target data source.

[0021] In step S102, according to a preset strategy or adaptive sequence, corresponding probe commands are sent to the target data source to collect telemetry data, wherein the sending timestamp Tcmd of each probe command is recorded.

[0022] In step S103, the raw byte streams from the corresponding data sources are received and buffered, and a arrival timestamp Tdata is applied to each raw byte stream.

[0023] In step S104, for any sent probe command, candidate response data blocks are located in the original byte stream based on its sending timestamp and a preset response delay window [Tcmd+ΔTmin, Tcmd+ΔTmax]. For example, one or more candidate response data blocks are located. ΔTmin and ΔTmax are the minimum and maximum response delays, respectively, and can be set according to the communication link delay and device response status. After locating the candidate response data blocks, the following steps are taken: selecting a response transformation model that matches the candidate response data blocks from the preset probe command-response transformation model mapping relationship.

[0024] In step S105, based on the dynamic decoding rule base, corresponding candidate decoding rules are generated for any candidate response data block, and trial decoding is performed using the candidate decoding rules. In some embodiments, the mapping relationship encapsulates response modes with different data encodings, and the response modes include:

[0025] In a counter model, the value of a field in a response data block increments with the instruction sequence number.

[0026] The status bit mapping model maps instruction-triggered status changes, such as "on" or "off," to the toggling of specified bits or groups in a data block.

[0027] In a parametric linear encoding model, numerical parameters contained in the instruction are scaled and appear at specific locations within a data block; for example, linear scaling y=a. x+b, where y is the scaled value, a is the scaling factor, x is the initial value, and b is the scaling basis. Each combination of "candidate response data block + response transformation model" generates a candidate decoding rule. This rule defines how to extract data from a specific offset of the original byte and what transformation (such as the formula y=(xb) / a) to apply to restore it to a physical quantity.

[0028] In step S106, a rationality assessment is performed based on the physical quantity data sequence output by the trial decoding, and the applicable decoding rules for the telemetry data stream are determined based on the assessment results.

[0029] This application employs a reverse parsing method of actively sending probing commands, observing responses, and performing correlation analysis. This method can purposefully induce regular changes in the data stream, thereby providing anchorable and causally related characteristic signals for parsing. It enables the systematic extraction of potential decoding rules from encrypted byte streams that are traditionally considered noise.

[0030] In some embodiments, generating corresponding candidate decoding rules for any candidate response data block includes:

[0031] For each combination of trial instruction-response transformation model, a structured candidate decoding rule description is generated. The candidate decoding rule description includes the identifier of the data source to which the rule applies, the byte offset of the rule's effect, and the algorithm for data extraction and transformation. For example, physical quantity = (byte [4:5] - offset). The proportionality coefficient, and the initial confidence value for generating the rule, which can be calculated based on the matching similarity.

[0032] The candidate decoding rules are cached and tested sequentially to output a sequence of physical quantities.

[0033] The reasonableness of the physical quantity sequence output from the trial decoding is verified, and a reasonableness score is given to the candidate decoding rules based on the verification results. In some embodiments, the reasonableness assessment based on the physical quantity data sequence output from the trial decoding includes employing at least one of the following physical quantity constraints:

[0034] Continuity constraints describe physical quantities that do not undergo abrupt changes.

[0035] Range constraints describe the range within which a physical quantity exists;

[0036] Causal consistency constraints describe the correspondence between physical quantities and equipment states.

[0037] Candidate decoding rules with a reasonableness score greater than the specified score threshold are used as decoding rules and the decoding is performed.

[0038] In some embodiments, it also includes:

[0039] Candidate decoding rules whose reasonableness scores exceed the first threshold are added to the dynamic decoding rule base;

[0040] For candidate decoding rules whose reasonableness scores fall between the second and first thresholds, the rules' parameters are adjusted and they are re-validated. In a specific example, the first and second thresholds can be set according to actual needs.

[0041] The rules in the dynamic decoding rule base are clustered. Based on the clustering results, the rule parameter with the most overlapping verification data is used as the master node, and a tree branch is constructed. The leaf nodes of the tree branch are constructed according to the amount of overlapping verification data rule parameters. Non-overlapping rule parameters are used as terminal nodes. In a specific example, the more overlapping verification data in progressive verification, the more likely it is to become the master node.

[0042] In some embodiments, clustering the rules in the dynamic decoding rule base includes:

[0043] Periodically, or when the number of effective rule sets exceeds a preset number, the effective rules in the dynamic decoding rule base are clustered. The clustering criteria include:

[0044] Scope overlap: Whether the byte offset ranges of the rules overlap or are adjacent;

[0045] Semantic relevance: whether the physical quantities corresponding to the rules belong to the same subsystem, such as batteries, transformers, oil tanks, etc.

[0046] Whether the algorithm model used by the rules is the same when changing the model type, such as when they are both linear models.

[0047] In some embodiments, constructing a tree-like branch includes:

[0048] Based on the clustering results, construct a tree-like branch structure for each type of rule;

[0049] Within the same cluster, the rule with the largest amount of validation data and the most stable score is taken as the master node of that branch;

[0050] For other rules within the cluster, tree-like branch paths are constructed based on the degree of difference between their rule parameters and the master node, as well as the amount of overlap in their respective validation data. For example, rules with small differences and highly overlapping validation data are used as direct child nodes or substitute nodes of the master node, forming shorter paths.

[0051] Rules that do not overlap with the validation data of other rules within the cluster and are inferred to be valid based on trial instructions or context are attached as terminal nodes to the end of the branch. These nodes represent alternative rules to be fully validated.

[0052] In some embodiments, generating corresponding candidate decoding rules for any candidate response data block includes:

[0053] Upon receiving a new data packet, one or more applicable tree branches are determined through rapid matching, such as trying common frame headers.

[0054] Based on a tree-like branching structure, the main node sequentially performs trial decoding based on the corresponding rules of each node; if the main node fails, it tries the direct child node, and so on. This greatly reduces the number of invalid attempts and improves decoding speed.

[0055] The tree structure of this application describes the conflict between rules by placing nodes on different branches or on the same branch. It can monitor the frequency and score of activation of nodes on different branches for the same data segment. Nodes that are active for a long time and have high scores can replace the original master node, achieving a smooth evolution of rules.

[0056] The results of the statistical decoding are analyzed, and the tree-like branch structure is maintained periodically based on the results.

[0057] As new rules are continuously added to the database and verification data accumulates, the rule tree is regularly reconstructed and optimized to ensure that its structure reflects the latest cognitive state.

[0058] When a node in a rule tree branch is found to be continuously failing or its score is decreasing during parsing, or when context verification discovers a new abnormal pattern, two training methods are triggered:

[0059] Local: For this abnormal pattern, plan specific probing instructions and generate new candidate rules to correct or supplement the branch.

[0060] Global: When the anomaly is widespread, a new round of extensive exploration is triggered to discover potentially entirely new protocol features, generating new rule clusters and tree-like branches.

[0061] This application introduces a clustering-based self-organizing and tree-like management structure for the rule base, achieving systematization of rule knowledge and a significant leap in parsing efficiency. Building upon dynamically generated and validated rules, the effective rule set is further intelligently clustered, and a tree-like structure is constructed with the most reliable rules as the main nodes, and branches organized according to parameter differences and the correlation of validation data. This allows the dynamic decoding rule base to evolve from an unordered set into an ordered knowledge system. This application not only solves the management and conflict resolution problems arising from an increasing number of rules, but also significantly reduces the computational overhead of real-time parsing by providing an efficient hierarchical retrieval path during parsing, achieving a balance between parsing speed and accuracy in complex and unknown protocol environments.

[0062] This application also proposes a telemetry data stream parsing system based on a dynamic decoding rule base, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the telemetry data stream parsing method based on the aforementioned dynamic decoding rule base.

[0063] Furthermore, although exemplary embodiments have been described herein, their scope includes any and all embodiments based on this disclosure that have equivalent elements, modifications, omissions, combinations (e.g., schemes involving intersections of various embodiments), adaptations, or changes. They are not limited to the examples described in this specification or during the implementation of this application, and such examples are to be construed as non-exclusive.

[0064] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. Other embodiments can be used by those skilled in the art when reading the above description.

[0065] The above embodiments are merely exemplary embodiments of this disclosure. Those skilled in the art can make various modifications or equivalent substitutions to this invention within the scope of the disclosure, and such modifications or equivalent substitutions should also be considered to fall within the protection scope of this invention.

Claims

1. A telemetry data stream parsing method based on a dynamic decoding rule base, characterized in that, include: Construct a dynamic decoding rule base and a trial instruction set, wherein the dynamic decoding rule base initially contains general decoding meta-rules, and the trial instruction set contains instructions that produce observable physical effects on the state of the target data source; According to a preset strategy or adaptive sequence, send corresponding probe commands to the target data source and collect telemetry data, recording the sending timestamp of each probe command. Receive and buffer raw byte streams from the corresponding data source, and add an arrival timestamp to each raw byte stream; For any sent probe command, the candidate response data block is located in the raw byte stream based on its sending timestamp and the preset response delay window; Based on the dynamic decoding rule base, corresponding candidate decoding rules are generated for any candidate response data block, and trial decoding is performed using the candidate decoding rules; A rationality assessment is conducted based on the physical quantity data sequence output from the trial decoding, and the applicable decoding rules for the telemetry data stream are determined based on the assessment results. Generating corresponding candidate decoding rules for any candidate response data block includes: For each combination of trial instruction-response transformation model, a structured candidate decoding rule description is generated. The candidate decoding rule description includes the identifier of the data source to which the rule applies, the byte offset of the rule's effect, the data extraction and transformation algorithm, and the initial confidence value for generating the rule. The candidate decoding rules are cached and trial decoding is performed one by one to output the physical quantity sequence; The rationality of the physical quantity sequence output by the trial decoding is verified, and the rationality score of the candidate decoding rule is given based on the verification results; Candidate decoding rules with a reasonableness score greater than the specified score threshold are used as decoding rules and the decoding is performed. Also includes: Candidate decoding rules whose reasonableness scores exceed the first threshold are added to the dynamic decoding rule base; For candidate decoding rules whose reasonableness scores fall between the second and first thresholds, adjust the rule parameters and re-validate them; The rules in the dynamic decoding rule base are clustered. The rule parameter with the most overlapping part of the verification data in the clustering results is used as the master node to construct a tree branch. The leaf nodes of the tree branch are constructed according to the amount of overlapping rule parameters in the verification data. The non-overlapping rule parameters are used as the terminal nodes.

2. The telemetry data stream parsing method based on a dynamic decoding rule base as described in claim 1, characterized in that, After locating the candidate response data block, the following is included: From the preset mapping relationship of the trial instruction-response transformation model, a response transformation model that matches the candidate response data block is selected. The mapping relationship encapsulates response modes with different data encodings, and the response modes include: In a counter model, the value of a field in a response data block increments with the instruction sequence number. The status bit mapping model maps instruction-triggered status changes to the toggling of specified bits or groups in a data block. In a parametric linear encoding model, the numerical parameters contained in the instruction are scaled and appear at specific locations in the data block.

3. The telemetry data stream parsing method based on a dynamic decoding rule base as described in claim 1, characterized in that, Clustering the rules in the dynamic decoding rule base includes: Periodically, or when the number of effective rule sets exceeds a preset number, the effective rules in the dynamic decoding rule base are clustered. The clustering criteria include: Scope overlap: Whether the byte offset ranges of the rules overlap or are adjacent; Semantic relevance: whether the physical quantities corresponding to the rules belong to the same subsystem; Whether the algorithm model used for the rule is the same regardless of the model type.

4. The telemetry data stream parsing method based on a dynamic decoding rule base as described in claim 3, characterized in that, Constructing tree branches includes: Based on the clustering results, construct a tree-like branch structure for each type of rule; Within the same cluster, the rule with the largest amount of validation data and the most stable score is taken as the master node of that branch; For other rules within the cluster, the path of the tree branch is constructed based on the degree of difference between their rule parameters and the master node, as well as the amount of overlap in their respective validation data. Rules that do not overlap with the validation data of other rules within the cluster and are inferred to be valid based on trial instructions or context are attached to the end of the branch as terminal nodes.

5. The telemetry data stream parsing method based on a dynamic decoding rule base as described in claim 3, characterized in that, Generating corresponding candidate decoding rules for any candidate response data block includes: Upon receiving a new data packet, one or more applicable tree branches are determined through quick matching; Based on the tree-branched main node, trial decoding is performed sequentially according to the corresponding rules of each node; The results of the statistical decoding are analyzed, and the tree-like branch structure is maintained periodically based on the results.

6. The telemetry data stream parsing method based on a dynamic decoding rule base as described in claim 3, characterized in that, The rationality assessment based on the physical quantity data sequence output by the trial decoding includes adopting at least one of the following physical quantity constraints: Continuity constraints describe physical quantities that do not undergo abrupt changes. Range constraints describe the range within which a physical quantity exists; Causal consistency constraints describe the correspondence between physical quantities and equipment states.

7. A telemetry data stream parsing system based on a dynamic decoding rule base, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the telemetry data stream parsing method based on a dynamic decoding rule base as described in any one of claims 1 to 6.

Images