Dynamic decoding rule base-based telemetry data stream analysis method and system

Abstract

The invention discloses a telemetering data stream analysis method and system based on a dynamic decoding rule base, and relates to the telemetering and data processing technology, and the method comprises the steps: constructing a dynamic decoding rule base and a tentative instruction set; according to a preset strategy or an adaptive sequence, sending a corresponding tentative instruction to the target data source, and collecting telemetering data; receiving and caching an original byte stream from a corresponding data source; for any sent heuristic instruction, in the original byte stream, according to a sending timestamp and a preset response delay window of the heuristic instruction, positioning a candidate response data block; based on the dynamic decoding rule base, generating a corresponding candidate decoding rule for any candidate response data block, and performing trial decoding by using the candidate decoding rule; and carrying out rationality evaluation so as to determine an applicable decoding rule of the telemetry data stream according to an evaluation result. The invention provides an effective mechanism for actively and systematically guiding and automatically generating trusted decoding rules.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of telemetry and data processing, and particularly relates to a telemetry data stream analysis method and system based on a dynamic decoding rule library. BACKGROUND

[0002] Telemetry data stream analysis is a key technology in the field of industrial equipment monitoring, and its basic principle is to restore the received binary byte stream into parameters with actual physical meaning, such as temperature, voltage, attitude, etc., according to a pre-set, fixed decoding rule (usually referred to as frame format definition). Traditional analysis systems rely heavily on complete prior knowledge of the target device communication protocol. The decoding rule usually exists in the form of a static configuration file or hard coding, and the system analyzes according to the synchronization header, field length, offset, data type and calibration formula defined in the rule. This method is simple and effective in scenarios where the protocol is known and stable.

[0003] In many practical application scenarios, such as downlink data with private encryption, non-cooperative target monitoring, and historical system integration, the telemetry data stream encountered in the analysis is often “black box” data with unknown protocol and encryption or complex coding. In the face of such data streams, the traditional static rule analysis method is completely ineffective. Therefore, some reverse engineering or blind analysis methods have appeared in the prior art, trying to infer the structure of the data stream itself. Common methods rely on statistical analysis of static data sets, for example, finding a fixed byte sequence that appears repeatedly as a potential frame header or frame trailer, or distinguishing data from padding fields through entropy analysis.

[0004] The prior art relies heavily on a large number of sample data, and it is difficult to work when sample data is scarce in the initial stage; secondly, when the data stream is strongly encrypted or obfuscated, its statistical characteristics tend to be random noise, making the recognition method based on statistical patterns extremely inefficient or even completely failed, and the existing method can only make a shallow speculation of the data format, cannot accurately map the binary segment to a specific physical quantity and its unit of measurement, and cannot verify the correctness of the mapping relationship. SUMMARY

[0005] The embodiments of the present application provide a telemetry data stream analysis method and system based on a dynamic decoding rule library, and propose an effective mechanism for actively, systematically guiding and automatically generating trusted decoding rules.

[0006] The embodiments of the present application propose a telemetry data stream analysis method based on a dynamic decoding rule library, comprising: constructing a dynamic decoding rule library and a heuristic instruction set, wherein the dynamic decoding rule library initially contains general decoding element rules, and the heuristic instruction set contains instructions that have an observable physical impact on the state of the target data source; sending corresponding tentative instructions to the target data source according to a preset strategy or an adaptive sequence, collecting telemetry data, wherein a sending time stamp of each tentative instruction is recorded; receiving and buffering original byte streams from the corresponding data source, and setting an arrival time stamp for each original byte stream; for any sent tentative instruction, locating a candidate response data block in the original byte stream according to the sending time stamp and a preset response delay window; generating a corresponding candidate decoding rule for any candidate response data block based on the dynamic decoding rule library, and performing tentative decoding by using the candidate decoding rule; performing rationality evaluation according to the physical quantity data sequence output by the tentative decoding, and determining the decoding rule of the telemetry data stream according to the evaluation result.

[0007] The application also provides a telemetry data stream analysis system based on a dynamic decoding rule library, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to realize the telemetry data stream analysis method based on the dynamic decoding rule library as described above.

[0008] The application can induce regular changes in the data stream by actively sending tentative instructions, observing responses, and performing reverse analysis, thereby providing anchoring feature signals with a cause-and-effect relationship, and realizing systematic stripping of potential decoding rules from encrypted byte streams that are regarded as noise in traditional methods.

[0009] The above description is only a summary of the technical solutions of the application. In order to more clearly understand the technical means of the application, the application can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the application more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0010] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Moreover, the same reference numerals in the attached drawings indicate the same or similar components. In the drawings: Figure 1 The basic flow of the telemetry data stream analysis method based on the dynamic decoding rule library of the present embodiment is shown in the figure. DETAILED DESCRIPTION

[0011] Exemplary embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thoroughly and completely understood, and will fully convey the scope of the present disclosure to those skilled in the art.

[0012] The present application proposes a telemetry data stream analysis method based on dynamic decoding rule base, as shown in Figure 1 The method comprises the following steps: In step S101, a dynamic decoding rule base and a heuristic instruction set are constructed, wherein the dynamic decoding rule base initially contains general decoding element rules, or can be initially empty, and the heuristic instruction set contains instructions that have observable physical effects on the state of the target data source.

[0013] In step S102, corresponding heuristic instructions are sent to the target data source according to a preset strategy or an adaptive sequence, and telemetry data is collected, wherein the sending time stamp Tcmd of each heuristic instruction is recorded.

[0014] In step S103, the original byte stream from the corresponding data source is received and cached, and an arrival time stamp Tdata is added to each original byte stream.

[0015] In step S104, for any sent heuristic instruction, in the original byte stream, according to its sending time stamp and a preset response delay window [Tcmd+ΔTmin, Tcmd+ΔTmax], a candidate response data block is located, for example, one or more candidate response data blocks are located. ΔTmin and ΔTmax are the minimum response delay and the maximum response delay, respectively, which can be set according to the communication link delay and the device response state. After locating the candidate response data block, the following steps are included: selecting a response transformation model that is adapted to the candidate response data block from the mapping relationship of the preset heuristic instruction-response transformation model.

[0016] In step S105, based on the dynamic decoding rule base, a corresponding candidate decoding rule is generated for any candidate response data block, and the candidate decoding rule is used for trial decoding. In some embodiments, the mapping relationship encapsulates reaction patterns of different data encoding, and the reaction patterns include: A counter model, in which the value of a certain field in the response data block increases with the instruction sequence number; A state bit mapping model, in which the state change triggered by the instruction, such as "on" / "off", is mapped to the flip of a specified bit or group in the data block; A parameter linear encoding model, in which a numerical parameter contained in the instruction appears in a specific position of the data block after scaling, for example, linear scaling y=a x+b, where y is the scaled value, a is the scaling factor, x is the initial value, and b is the scaling basis. Each combination of "candidate response data block + response transformation model" generates a candidate decoding rule. This rule defines how to extract data from a specific offset of the original byte and what transformation (such as the formula y=(xb) / a) to apply to restore it to a physical quantity.

[0017] In step S106, a rationality assessment is performed based on the physical quantity data sequence output by the trial decoding, and the applicable decoding rules for the telemetry data stream are determined based on the assessment results.

[0018] This application employs a reverse parsing method of actively sending probing commands, observing responses, and performing correlation analysis. This method can purposefully induce regular changes in the data stream, thereby providing anchorable and causally related characteristic signals for parsing. It enables the systematic extraction of potential decoding rules from encrypted byte streams that are traditionally considered noise.

[0019] In some embodiments, generating corresponding candidate decoding rules for any candidate response data block includes: For each combination of trial instruction-response transformation model, a structured candidate decoding rule description is generated. The candidate decoding rule description includes the identifier of the data source to which the rule applies, the byte offset of the rule's effect, and the algorithm for data extraction and transformation. For example, physical quantity = (byte [4:5] - offset). The proportionality coefficient, and the initial confidence value for generating the rule, which can be calculated based on the matching similarity.

[0020] The candidate decoding rules are cached and tested sequentially to output a sequence of physical quantities.

[0021] The reasonableness of the physical quantity sequence output from the trial decoding is verified, and a reasonableness score is given to the candidate decoding rules based on the verification results. In some embodiments, the reasonableness assessment based on the physical quantity data sequence output from the trial decoding includes employing at least one of the following physical quantity constraints: Continuity constraints describe physical quantities that do not undergo abrupt changes. Range constraints describe the range within which a physical quantity exists; Causal consistency constraints describe the correspondence between physical quantities and equipment states.

[0022] Candidate decoding rules with a reasonableness score greater than the specified score threshold are used as decoding rules and the decoding is performed.

[0023] In some embodiments, it also includes: Candidate decoding rules whose reasonableness scores exceed the first threshold are added to the dynamic decoding rule base; For candidate decoding rules whose reasonableness scores fall between the second and first thresholds, the rules' parameters are adjusted and they are re-validated. In a specific example, the first and second thresholds can be set according to actual needs.

[0024] The rules in the dynamic decoding rule base are clustered. Based on the clustering results, the rule parameter with the most overlapping verification data is used as the master node, and a tree branch is constructed. The leaf nodes of the tree branch are constructed according to the amount of overlapping verification data rule parameters. Non-overlapping rule parameters are used as terminal nodes. In a specific example, the more overlapping verification data in progressive verification, the more likely it is to become the master node.

[0025] In some embodiments, clustering the rules in the dynamic decoding rule base includes: Periodically, or when the number of effective rule sets exceeds a preset number, the effective rules in the dynamic decoding rule base are clustered. The clustering criteria include: Scope overlap: Whether the byte offset ranges of the rules overlap or are adjacent; Semantic relevance: whether the physical quantities corresponding to the rules belong to the same subsystem, such as batteries, transformers, oil tanks, etc.

[0026] Whether the algorithm model used by the rules is the same when changing the model type, such as when they are both linear models.

[0027] In some embodiments, constructing a tree-like branch includes: Based on the clustering results, construct a tree-like branch structure for each type of rule; Within the same cluster, the rule with the largest amount of validation data and the most stable score is taken as the master node of that branch; For other rules within the cluster, tree-like branch paths are constructed based on the degree of difference between their rule parameters and the master node, as well as the amount of overlap in their respective validation data. For example, rules with small differences and highly overlapping validation data are used as direct child nodes or substitute nodes of the master node, forming shorter paths.

[0028] Rules that do not overlap with the validation data of other rules within the cluster and are inferred to be valid based on trial instructions or context are attached as terminal nodes to the end of the branch. These nodes represent alternative rules to be fully validated.

[0029] In some embodiments, generating corresponding candidate decoding rules for any candidate response data block includes: Upon receiving a new data packet, one or more applicable tree branches are determined through rapid matching, such as trying common frame headers.

[0030] Based on a tree-like branching structure, the main node sequentially performs trial decoding based on the corresponding rules of each node; if the main node fails, it tries the direct child node, and so on. This greatly reduces the number of invalid attempts and improves decoding speed.

[0031] The tree structure of this application describes the conflict between rules by placing nodes on different branches or on the same branch. It can monitor the frequency and score of activation of nodes on different branches for the same data segment. Nodes that are active for a long time and have high scores can replace the original master node, achieving a smooth evolution of rules.

[0032] The results of the statistical decoding are analyzed, and the tree-like branch structure is maintained periodically based on the results.

[0033] As new rules are continuously added to the database and verification data accumulates, the rule tree is regularly reconstructed and optimized to ensure that its structure reflects the latest cognitive state.

[0034] When a node in a rule tree branch is found to be continuously failing or its score is decreasing during parsing, or when context verification discovers a new abnormal pattern, two training methods are triggered: Local: For this abnormal pattern, plan specific probing instructions and generate new candidate rules to correct or supplement the branch.

[0035] Global: When the anomaly is widespread, a new round of extensive exploration is triggered to discover potentially entirely new protocol features, generating new rule clusters and tree-like branches.

[0036] This application introduces a clustering-based self-organizing and tree-like management structure for the rule base, achieving systematization of rule knowledge and a significant leap in parsing efficiency. Building upon dynamically generated and validated rules, the effective rule set is further intelligently clustered, and a tree-like structure is constructed with the most reliable rules as the main nodes, and branches organized according to parameter differences and the correlation of validation data. This allows the dynamic decoding rule base to evolve from an unordered set into an ordered knowledge system. This application not only solves the management and conflict resolution problems arising from an increasing number of rules, but also significantly reduces the computational overhead of real-time parsing by providing an efficient hierarchical retrieval path during parsing, achieving a balance between parsing speed and accuracy in complex and unknown protocol environments.

[0037] This application also proposes a telemetry data stream parsing system based on a dynamic decoding rule base, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the telemetry data stream parsing method based on the aforementioned dynamic decoding rule base.

[0038] Furthermore, although exemplary embodiments have been described herein, their scope includes any and all embodiments based on this disclosure that have equivalent elements, modifications, omissions, combinations (e.g., schemes involving intersections of various embodiments), adaptations, or changes. They are not limited to the examples described in this specification or during the implementation of this application, and such examples are to be construed as non-exclusive.

[0039] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. Other embodiments can be used by those skilled in the art when reading the above description.

[0040] The above embodiments are merely exemplary embodiments of this disclosure. Those skilled in the art can make various modifications or equivalent substitutions to this invention within the scope of the disclosure, and such modifications or equivalent substitutions should also be considered to fall within the protection scope of this invention.

Claims

1. A method for telemetry data stream parsing based on dynamic decoding rule base, characterized in that, The method comprises the following steps: constructing a dynamic decoding rule base and a heuristic instruction set, wherein the dynamic decoding rule base initially contains general decoding element rules, and the heuristic instruction set contains instructions that have observable physical effects on the target data source state; sending corresponding heuristic instructions to the target data source according to a preset strategy or an adaptive sequence, collecting telemetry data, and recording the sending time stamp of each heuristic instruction; receiving and buffering the original byte stream from the corresponding data source, and setting the arrival time stamp for each original byte stream; for any sent heuristic instruction, locating the candidate response data block in the original byte stream according to the sending time stamp and the preset response delay window; generating a corresponding candidate decoding rule for any candidate response data block based on the dynamic decoding rule base, and performing trial decoding using the candidate decoding rule; performing rationality evaluation according to the physical quantity data sequence output by the trial decoding, and determining the decoding rule of the telemetry data stream according to the evaluation result.

2. The dynamic decoding ruleset-based telemetry data stream parsing method of claim 1, wherein, After locating the candidate response data block, the following steps are included: selecting a response transformation model that is suitable for the candidate response data block from the mapping relationship of the preset heuristic instruction-response transformation model, wherein the mapping relationship encapsulates different data encoding reaction modes, and the reaction modes include: a counter model, in which the value of a certain field in the response data block increases with the instruction sequence number; a state bit mapping model, in which the state change triggered by the instruction is mapped to the flip of a specified bit or group in the data block; a parameter linear encoding model, in which the numerical parameter contained in the instruction appears in a specific position of the data block after scaling.

3. The dynamic decoding ruleset-based telemetry data stream parsing method of claim 2, wherein, Generating a corresponding candidate decoding rule for any candidate response data block includes: generating a structured candidate decoding rule description for each combination of heuristic instruction-response transformation models, which includes the identification of the data source to which the rule applies, the byte offset of the rule, the data extraction and transformation algorithm, and the initial confidence value of the rule; buffering the candidate decoding rule and performing trial decoding in sequence to output a physical quantity sequence; performing rationality verification on the physical quantity sequence output by the trial decoding to score the candidate decoding rule according to the verification result; selecting the candidate decoding rule with a rationality score greater than a specified score threshold as the decoding rule for execution.

4. The dynamic decoding ruleset-based telemetry data stream parsing method of claim 3, wherein, Further comprising: adding the candidate decoding rule with a rationality score exceeding a first threshold to the dynamic decoding rule base; adjusting the parameters of the candidate decoding rule with a rationality score between a second threshold and the first threshold, and re-verifying it; clustering the rules in the dynamic decoding rule base, selecting the rule parameter with the most overlapping verification data as the main node according to the clustering result, constructing a tree-shaped branch, wherein the leaf nodes of the tree-shaped branch are constructed according to the number of overlapping rule parameters of the verification data, and the non-overlapping rule parameters are selected as the end nodes.

5. The dynamic decoding ruleset-based telemetry data stream parsing method of claim 4, wherein, The clustering of the rules in the dynamic decoding rule base includes: periodically or when the number of active rules in the dynamic decoding rule base exceeds a preset number, clustering the active rules in the dynamic decoding rule base, and the clustering basis includes: scope overlap, whether the byte offset range of the rule action overlaps or is adjacent; Semantic relevance, whether the physical quantities corresponding to the rules belong to the same subsystem; Transformation model type, whether the algorithm models adopted by the rules are the same.

6. The dynamic decoding ruleset-based telemetry data stream parsing method of claim 5, wherein, The tree branch construction includes: According to the clustering results, a tree branch structure is constructed for each class of rules; In the same cluster, the rule with the most validation data and the most stable score is taken as the main node of the branch; For other rules in the cluster, according to the difference degree of their rule parameters and the main node, and how much their respective validation data overlap, the path of the tree branch is constructed; Rules that do not overlap with the validation data of other rules in the cluster and are effective according to the heuristic instructions or context speculation are taken as the end nodes and are mounted at the end of the branch.

7. The dynamic decoding ruleset-based telemetry data stream parsing method of claim 5, wherein, Generating a corresponding candidate decoding rule for any candidate response data block includes: After receiving a new data packet, one or several tree branches suitable for use are determined through fast matching; Based on the main node of the tree branch structure, the trial decoding is sequentially performed based on the corresponding rules of each node; The results of the trial decoding are counted, and the tree branch structure is regularly maintained according to the results.

8. The dynamic decoding ruleset-based telemetry data stream parsing method of claim 5, wherein, The rationality evaluation based on the physical quantity data sequence output by the trial decoding includes adopting at least one of the following physical quantity constraints: Continuity constraint, describing that the physical quantity does not jump; Value range constraint, describing that the physical quantity is within a set range; Causal consistency constraint, describing the corresponding relationship between the physical quantity and the device state.

9. A telemetry data stream parsing system based on a dynamic decoding rules library, characterized in that, A processor and a memory are included, and the memory has stored thereon a computer program, which, when executed by the processor, implements the telemetry data stream analysis method based on the dynamic decoding rule library according to any one of claims 1 to 8.

Images