Data processing method, system and device

Abstract

This invention provides a data processing method, system, and device. The method includes: obtaining the protocol type, data format, and structural features of the data stream to be processed based on its multidimensional data feature vector; constructing a target parsing rule base matching the data stream to be processed based on the protocol type, data format, and structural features; parsing the data stream to be processed according to the target parsing rule base to obtain target structure data; performing semantic enhancement and consistency verification on the target structure data; and outputting standardized parsed data of the data stream to be processed based on the enhanced target structure data and the consistency verification results. This invention achieves end-to-end automated conversion from raw heterogeneous data streams to high-quality standardized parsed data, significantly improving the intelligence level and processing efficiency of multi-source heterogeneous data access.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, system, and device. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), the Industrial Internet, and big data technologies, the data sources accessing systems exhibit significant multi-source heterogeneity, encompassing complex communication protocols, diverse data formats, and varied logical structures. Therefore, achieving standardized parsing and processing of massive heterogeneous data streams has become a key technological challenge supporting big data integration and intelligent decision-making systems.

[0003] To meet the above requirements, existing technical solutions typically use static parsing templates or hard-coded rules to parse data streams, converting the raw data streams into a preset structured format.

[0004] However, due to the fixed boundaries of the processing logic in existing technologies, they suffer from low processing efficiency, poor accuracy, and poor scalability when faced with ever-changing data characteristics and complex protocol forms. Summary of the Invention

[0005] This invention provides a data processing method, system, and device to solve the technical problems of low processing efficiency, poor processing accuracy, and poor scalability in the prior art.

[0006] This invention provides a data processing method, comprising: Based on the multidimensional data feature vector of the data stream to be processed, the protocol type, data format, and structural features of the data stream to be processed are obtained; Based on the protocol type, the data format, and the structural features, construct a target parsing rule base that matches the data stream to be processed; The target parsing rule base is used to parse the data stream to be processed to obtain the target structure data; The target structure data is semantically enhanced and consistency verified. Based on the enhanced target structure data and the consistency verification results, the standardized parsed data of the data stream to be processed is output.

[0007] According to a data processing method provided by the present invention, the step of constructing a target parsing rule base matching the data stream to be processed based on the protocol type, the data format, and the structural features includes: Based on the protocol type, the structural features, and the data format, load multiple historical parsing records that match the data stream to be processed into the parsing record library; Calculate the similarity of each of the historical parsed records in the field structure feature space; Based on the similarity, the basic parsing rule base is updated to obtain the target parsing rule base.

[0008] According to a data processing method provided by the present invention, the step of updating the basic parsing rule base based on the similarity to obtain the target parsing rule base includes: Aggregate target parsing records with a similarity greater than a preset clustering threshold from multiple historical parsing records into target parsing clusters; Extract the common parsing strategy among all target parsing records from the target parsing cluster; the common parsing strategy includes a common field mapping strategy and a common parsing path; Based on the shared parsing strategy, the basic parsing rule base is incrementally updated to obtain the target parsing rule base.

[0009] According to a data processing method provided by the present invention, the method further includes: When all similarities are less than or equal to the preset clustering threshold, the field mapping strategy of the data stream to be processed is inferred based on the protocol type, the data format, and the structural features. Generate an initial parsing rule template based on the field mapping strategy; Send the initial parsing rule template and the confidence label corresponding to the initial parsing rule template to the client; Receive the completion confirmation information of the initial parsing rule template returned by the client; Based on the completion confirmation information and the initial parsing rule template, the basic parsing rule base is incrementally updated to obtain the target parsing rule base.

[0010] According to a data processing method provided by the present invention, the target structured data includes a raw data layer, a metadata layer, and a semantic tag layer; The original data layer includes multiple components from the following: hierarchical structure information of the data stream to be processed, binary and text features, data packet boundary information, and frame positioning information. The metadata layer includes metadata description information organized in a key-value structure and a data quality score; the data quality score is obtained by weighted summation of field parsing completeness, field semantic matching confidence, and field value verification consistency; the metadata description information includes data source identifier, collection timestamp, protocol version number, and a field definition dictionary used to identify parsing result attributes; The semantic tagging layer includes structured semantic tags; the structured semantic tags are generated by using a multi-tag classifier to perform entity recognition, concept classification, and semantic relationship annotation on the field values ​​of each field in the data stream to be processed, based on the structural docking relationship between the metadata layer and the knowledge graph.

[0011] According to a data processing method provided by the present invention, the semantic enhancement of the target structure data includes: A heterogeneous attribute graph is constructed using the metadata description information and the structured semantic tags as graph nodes; Based on the weight coefficients corresponding to each edge type and the attention coefficients between each graph node, the representation vectors of each graph node in the heterogeneous attribute graph are updated, and the implicit associations between each field are obtained based on the update results. Based on the implicit association, semantic enhancement is performed on the target structure data to obtain the enhanced target structure data.

[0012] According to a data processing method provided by the present invention, the output step of the standardized parsed data includes: The priority coefficient of the data stream to be processed is determined based on the data source identifier, the collection timestamp, the protocol version number, and the data quality score. Based on the priority coefficient, and the queuing time, actual processing time and maximum tolerable latency of the data stream to be processed under each candidate resource scheduling strategy, the target resource scheduling strategy is determined among the multiple candidate resource scheduling strategies. According to the target resource scheduling strategy, the parsing engine performs a resource reallocation action, and using the parsing engine after resource reallocation, based on the enhanced target structure data and the consistency verification result, outputs the standardized parsed data of the data stream to be processed to the data receiving end.

[0013] According to a data processing method provided by the present invention, the step of obtaining the protocol type, data format, and structural features of the data stream to be processed based on the multidimensional data feature vector of the data stream to be processed includes: For each dimension of data feature vector, calculate the dimensional feature similarity between the data stream to be processed and each candidate protocol parsing template in the parsing template library; Based on the weight coefficients corresponding to the feature similarity of each dimension, the feature similarity of each dimension is weighted and summed, and then normalized to obtain the matching score between the data stream to be processed and each of the candidate protocol parsing templates. Among the multiple candidate protocol parsing templates, the candidate protocol parsing templates with matching scores greater than a preset score threshold are selected. If there are multiple candidate protocol parsing templates, then the target protocol parsing template with the highest matching score is selected from the multiple candidate protocol parsing templates; Extract the protocol type, data format, and structural features of the data stream to be processed from the target protocol parsing template.

[0014] According to a data processing method provided by the present invention, the step of obtaining the multidimensional data feature vector of the data stream to be processed includes: Multiple data packets are extracted from the data stream to be processed, and the header information of the multiple data packets is aggregated by window sliding to obtain aggregated header information; Based on the aggregated header information, character sequence statistical features and field weight distribution features are extracted from the data stream to be processed to obtain a multidimensional data feature vector of the data stream to be processed; the multidimensional data feature vector includes multiple features such as field length distribution features, separator frequency features, field pattern overlap features, and control character position features.

[0015] The present invention also provides a data processing system, comprising: The dynamic protocol identification and adaptation module is used to obtain the protocol type, data format and structural features of the data stream to be processed based on the multi-dimensional data feature vector of the data stream to be processed; An adaptive parsing rule generation module is used to construct a target parsing rule library that matches the data stream to be processed based on the protocol type, the data format, and the structural features. A unified intermediate format conversion module is used to parse the data stream to be processed according to the target parsing rule base to obtain target structure data; The metadata enhancement and verification module is used to perform semantic enhancement and consistency verification on the target structure data; The standardized output module is used to output standardized parsed data of the data stream to be processed based on the enhanced target structure data and the consistency verification results.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data processing methods described above.

[0019] The data processing method, system, and device provided by this invention achieve end-to-end automated conversion from raw heterogeneous data streams to high-quality standardized parsed data through end-to-end processing of the data stream to be processed, including multi-dimensional data feature vector extraction and protocol attribute identification, adaptive parsing rule base construction, three-layer unified intermediate format conversion, semantic enhancement, and consistency verification. This effectively solves the problems of insufficient adaptability, loss of semantic information, and difficulty in guaranteeing data quality in traditional parsing technologies under multi-protocol environments, and significantly improves the intelligence level and processing efficiency of multi-source heterogeneous data access. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the data processing method provided by the present invention.

[0022] Figure 2 This is a schematic diagram of the process for generating the target parsing rule base provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the hierarchical structure of the target structure data provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the resource scheduling process provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the data processing system provided by the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] All actions involving the acquisition of signal information or data in this application are carried out in accordance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0029] With the rapid development of IoT, Industrial Internet, and big data technologies, data sources are becoming increasingly diverse and complex in format, encompassing text, binary streams, structured and semi-structured data. The continuous iteration of data protocols has rendered traditional single-protocol parsing solutions inadequate for meeting the demands for efficient and flexible data access. The efficient integration and semantic unification of heterogeneous data have become key technological challenges supporting intelligent analysis and decision-making systems.

[0030] Traditional technologies primarily rely on predefined static parsing templates or hard-coded protocol conversion rules, typically designed for a single or limited number of protocols, lacking adaptability to protocol changes. The parsing process often employs batch processing, performing hard format parsing first and then mapping to a unified format. Parsing rules are frequently updated and require manual maintenance. Furthermore, data integrity verification and semantic enhancement methods are limited, making it difficult to achieve cross-protocol semantic fusion and structural compatibility. In addition, system resource scheduling is mostly statically allocated, lacking dynamic response mechanisms to real-time load and priority, leading to performance bottlenecks and resource waste.

[0031] Due to limitations of traditional technologies, existing solutions struggle to guarantee high-accuracy protocol identification and parsing rule coverage in multi-protocol environments. They also cannot achieve automatic evolution and optimization of parsing rules, resulting in high costs and long response cycles when adapting to new data sources. Furthermore, the lack of a unified intermediate format structure and semantic enhancement capabilities leads to insufficient data correlation mining, directly impacting the quality of subsequent analysis and decision-making. In addition, static resource scheduling mechanisms are insufficient to meet the real-time parsing demands of high concurrency and multi-task operations, resulting in inadequate system stability and scalability. Therefore, there is an urgent need for a low-level data acquisition and parsing system with dynamic adaptation, incremental learning, semantic enhancement, and elastic resource scheduling capabilities to comprehensively improve the intelligence level and operational efficiency of heterogeneous data processing.

[0032] To address this issue, this application proposes a data processing method aimed at solving the problem that traditional data parsing techniques are ill-suited to multi-protocol and multi-format data environments. This method can automatically identify and adaptively parse various complex protocols, dynamically generate and optimize parsing rules, construct a unified intermediate format with semantic enhancement, strengthen data correlation analysis through graph neural networks, and improve the processing accuracy, efficiency, and scalability of the data processing system in high-concurrency and variable data environments by combining a dynamic scheduling mechanism for elastic resources. This effectively solves the shortcomings of traditional parsing techniques in heterogeneous data access, semantic fusion, and real-time processing, providing a high-quality data foundation for intelligent big data applications.

[0033] Figure 1This is a flowchart illustrating the data processing method provided by the present invention; the execution entity of this method is a data processing system, which is a low-level acquisition and parsing system. This method is mainly applied to the parsing of multi-source heterogeneous data in complex network environments, particularly addressing the heterogeneous data stream processing needs in scenarios such as industrial control, the Internet of Things, and financial transactions. Figure 1 As shown, the method includes the following steps: Step 110: Obtain the protocol type, data format, and structural features of the data stream to be processed based on the multidimensional data feature vector of the data stream to be processed.

[0034] Optionally, during the parsing and processing of the required data stream, dynamic protocol identification and adaptation can be performed on the data stream first.

[0035] Specifically, a pre-built metadata learning engine can be used to collect any raw data stream from multiple heterogeneous data sources in real time, that is, the data stream to be processed.

[0036] The data stream to be processed here refers to the raw data payload that has not undergone standardization processing and is reported in real time by devices such as distributed edge nodes, sensors or business terminals. Different data streams to be processed have significant heterogeneity in physical form and logical encoding. They can be text data in the form of character sequences, binary stream data in the form of bit streams, or structured or semi-structured data with specific organization.

[0037] Because these data streams come from a wide range of sources and use diverse protocols, direct parsing often faces difficulties due to format mismatches. Therefore, it is necessary to first extract multi-dimensional features from them in order to obtain multi-dimensional data feature vectors that can characterize their communication patterns and message attributes.

[0038] The multidimensional data feature vector here refers to a set of quantitative parameters extracted from the data stream to be processed, which can characterize its communication patterns and message attributes. It covers the feature information of the data stream in the dimensions of statistics, information theory and time domain distribution. It can be extracted by one or more machine learning techniques, such as character sequence statistical techniques and / or field weight distribution techniques. This embodiment does not specifically limit this.

[0039] For example, to ensure zero-configuration automatic adaptation, in one possible embodiment, the step of obtaining the multidimensional data feature vector of the data stream to be processed includes: Multiple data packets are extracted from the data stream to be processed, and the header information of the multiple data packets is aggregated by window sliding to obtain aggregated header information; Based on the aggregated header information, character sequence statistical features and field weight distribution features are extracted from the data stream to be processed to obtain a multidimensional data feature vector of the data stream to be processed; the multidimensional data feature vector includes multiple features such as field length distribution features, separator frequency features, field pattern overlap features, and control character position features.

[0040] Optionally, when extracting multidimensional data feature vectors from the data stream to be processed, the first N data packets are extracted from the real-time access data stream. The value of N can be dynamically adjusted according to the accuracy requirements of protocol recognition to balance recognition accuracy and system response efficiency. Then, the header information of the first N data packets is aggregated using a sliding window method to obtain aggregated header information. The sliding window method here refers to using a sliding window of preset length to scan the original payload of multiple data packets in a fixed or overlapping step order, thereby capturing the continuous features of the data in the time domain. The aggregated header information refers to the global feature set formed by logically recombining key bits such as protocol headers, sequence numbers, and control commands in discrete data packets through the above window scanning process. This information can effectively shield the random noise of a single message during transmission.

[0041] Furthermore, a pre-built metadata learning engine is used to initially determine the field patterns of the data stream to be processed. Here, field patterns refer to the physical arrangement of fields in the data stream, such as whether they are fixed-length fields or variable-length fields segmented by specific delimiters. Simultaneously, contextual semantic information of the data stream to be processed is extracted; this contextual semantic information describes the logical relationships or state transition patterns between data packets.

[0042] After obtaining the above information, a multi-feature extraction mechanism is introduced. This involves aggregating header information, field patterns, and contextual semantic information as prior guidance. The Term Frequency-Inverse Document Frequency (TF-IDF) weight distribution function is used to extract character sequence statistical features from the data stream to be processed. A character-level N-gram statistical model is used to extract field weight distribution features from the data stream. Through these two feature extraction techniques, a multi-dimensional data feature vector of the data stream to be processed is obtained. This multi-dimensional data feature vector includes, but is not limited to, multiple features such as field length distribution features, delimiter frequency features, field pattern overlap features, and control character position features.

[0043] Among them, field length distribution features refer to the statistical variation of the length of each semantic unit in the data stream to be processed, which is used to distinguish between fixed step size protocols and dynamic length protocols; delimiter frequency features refer to the frequency distribution of the occurrence of commas, semicolons or specific delimiters in the data stream, which is used to help delineate data boundaries; field pattern overlap features refer to the evaluation of the topological similarity between the structural arrangement of the current data stream and the known protocol paradigm; control character position features refer to the extraction of the absolute offset or relative position pattern of key protocol control codes in the message sequence.

[0044] Here, by employing a dynamic truncation and window sliding aggregation mechanism, combined with character-level statistics and weight distribution algorithms, a multi-dimensional feature vector with high discriminative power and strong robustness can be extracted from the original messy data stream. This not only significantly improves the breadth of recognition for unknown and complex protocols, but also balances recognition accuracy and system response efficiency by dynamically adjusting the sampling depth N, providing accurate data representation support for achieving zero-configuration adaptive access to heterogeneous data.

[0045] Once the multidimensional data feature vector of the data stream to be processed is obtained, the protocol type, data format, and structural characteristics of the data stream can be determined. This process can involve comparing the extracted feature vector with a pre-stored parsing template library to determine specific attributes, or directly performing inference and prediction on the multidimensional data feature vector to output its specific attributes. This embodiment does not specifically limit this approach.

[0046] For example, to improve robustness in identifying complex variant protocols, in one possible implementation, obtaining the protocol type, data format, and structural features of the data stream to be processed based on its multidimensional data feature vector includes: For each dimension of data feature vector, calculate the dimensional feature similarity between the data stream to be processed and each candidate protocol parsing template in the parsing template library; Based on the weight coefficients corresponding to the feature similarity of each dimension, the feature similarity of each dimension is weighted and summed, and then normalized to obtain the matching score between the data stream to be processed and each of the candidate protocol parsing templates. Among the multiple candidate protocol parsing templates, the candidate protocol parsing templates with matching scores greater than a preset score threshold are selected. If there are multiple candidate protocol parsing templates, then the target protocol parsing template with the highest matching score is selected from the multiple candidate protocol parsing templates; Extract the protocol type, data format, and structural features of the data stream to be processed from the target protocol parsing template.

[0047] Optionally, to ensure the system (i.e., the data processing system) has sufficient protocol recognition breadth and generalization capability, a parsing template library supporting 20 or more common industrial protocols, communication protocols, and structured data standards (such as Modbus, OPC-UA, BACnet, CAN, JSON, XML, Protobuf, etc.) as well as user-defined protocols can be pre-built. This library is built based on a metadata-driven mechanism, achieving automatic template parsing and loading through protocol layer definition, field semantic mapping, and data boundary description.

[0048] To further achieve zero-configuration automatic adaptation, a multi-feature extraction mechanism is introduced. Specifically, the TF-IDF weight distribution function is used to extract character sequence statistical features from the data stream to be processed, and the character-level N-gram statistical model is used to extract field weight distribution features from the data stream to be processed. Thus, a high-dimensional protocol feature vector, or multi-dimensional data feature vector, is obtained from the data stream to be processed.

[0049] Then, the matching score between the data stream to be processed and each candidate protocol parsing template is calculated using the following formula: ; in, Indicates candidate protocol parsing template Matching score between the data stream D to be processed; For the data feature vector in the j-th dimension, the candidate protocol parsing template The dimensional feature similarity between the data stream D to be processed, which includes field length distribution, delimiter frequency, field pattern overlap, control character position, etc. The weight coefficients are the feature similarity coefficients on the data feature vector of the j-th dimension, which can be automatically learned by a pre-trained model or a metadata engine based on the training set. Let be the dimension of the data feature vector. This formula, through a normalized weighted similarity calculation mechanism, strengthens the influence of dominant features in the protocol identification process, significantly improving the ability to distinguish highly similar protocols.

[0050] After obtaining the matching scores between the data stream to be processed and each candidate protocol parsing template, the candidate protocol parsing templates can be sorted in descending order of matching scores. The first candidate protocol parsing template in the sorted list will be selected when its matching score with the data stream to be processed is greater than a preset score threshold. When the target protocol is identified, the candidate protocol parsing template is loaded as the target protocol parsing template. A preset scoring threshold is used here. The default value can be set to 0.85. In practical applications, this threshold can be fine-tuned according to different balance requirements for protocol recognition accuracy and false negative rate.

[0051] If the matching score between all candidate protocol parsing templates and the data stream to be processed is lower than or equal to the preset score threshold. If the current data stream belongs to an unknown protocol or a variant protocol with extremely low similarity, it indicates that the data stream to be processed belongs to an unknown protocol or a variant protocol with very low similarity. In this case, it is necessary to automatically trigger the manual intervention interface or enter the custom rule configuration channel to obtain the protocol type, data format, and structural characteristics of the data stream to be processed. In the custom rule configuration channel, the system supports developers to build new protocol parsing templates using the Low-Code Rules Engine (LCRE) through a visual interface, define specific field boundaries, encoding formats, and conversion logic, thereby realizing the dynamic expansion and continuous learning of the protocol library.

[0052] It should be noted that the entire protocol identification and adaptation process can be driven by an embedded learning engine. This process not only ensures high identification accuracy in multi-source heterogeneous data environments, with testing showing that the accuracy can be consistently maintained above 98%, but also continuously optimizes the parameters of feature extraction operators and the weight coefficients in the matching formula through incremental model training. This dynamic identification mechanism ensures high adaptability and robustness when facing complex and variable data sources such as industrial edge nodes, IoT devices, and various customized control systems.

[0053] Step 120: Construct a target parsing rule base that matches the data stream to be processed, based on the protocol type, the data format, and the structural features.

[0054] Optionally, after accurately identifying the attribute information of the data stream to be processed, the adaptive parsing rule generation stage can be entered.

[0055] Specifically, after obtaining the protocol type, data format, and structural characteristics of the data stream to be processed, a target parsing rule base can be adaptively constructed based on these characteristics. In this embodiment, the target parsing rule base is a set of instructions used to guide the parsing engine in extracting, mapping, and logically transforming fields in the data stream to be processed. It includes, but is not limited to, field mapping relationships, data type conversion logic, and protocol conversion strategies. This embodiment does not specifically limit these aspects.

[0056] The target parsing rule base can be constructed in several ways, including: filtering target parsing records that match the data stream to be processed based on protocol type, data format, and structural features, and dynamically generating the target parsing rule base by utilizing the common parsing strategy among all target parsing records; or, using a structural similarity projection algorithm to heuristically infer the field mapping strategy of the data stream to be processed based on protocol type, data format, and structural features, thereby dynamically generating the target parsing rule base. The specific method can be dynamically determined based on the protocol type of the data stream to be processed.

[0057] For example, to achieve intelligent adaptive generation of parsing rules, in one possible embodiment, constructing a target parsing rule base matching the data stream to be processed based on the protocol type, the data format, and the structural features includes: Based on the protocol type, the structural features, and the data format, load multiple historical parsing records that match the data stream to be processed into the parsing record library; Calculate the similarity of each of the historical parsed records in the field structure feature space; Based on the similarity, the basic parsing rule base is updated to obtain the target parsing rule base.

[0058] Optionally, in the process of dynamically generating parsing rules, in order to achieve self-evolution and coverage enhancement of the parsing rule base, a clustering and induction method based on incremental learning mechanism can be introduced. By performing periodic structural similarity analysis on historical parsing records, structural feature vectors are automatically generated and multi-dimensional clustering is performed, thereby discovering potential protocol structure variations and data source heterogeneity patterns.

[0059] Figure 2 This is a schematic diagram of the process for generating the target parsing rule base provided by the present invention.

[0060] like Figure 2 As shown, firstly, multiple historical parsing records that match the protocol type, data format, structural characteristics, and other attribute information of the data stream to be processed are loaded from the parsing record library.

[0061] Then, structural similarity analysis is performed on each historical parsing record, that is, the similarity of each historical parsing record in the field structure feature space is calculated, specifically by the following similarity function: ; in, Represents historical analysis records Historical analysis records Similarity in the field structure feature space; This represents the frequency distribution function of the historical parsing record R on the k-th structure field. The structure field here includes field name pattern, field sequence position, data type distribution, nested hierarchy, etc. The number of structural fields represents the similarity function. This function is tolerant of structural ambiguity and highly sensitive to semantically preserved features, effectively supporting the classification and aggregation of protocol structural variation samples.

[0062] Then, multidimensional clustering is performed on multiple historical parsing records based on similarity. The basic parsing rule base is updated based on the clustering results to obtain the target parsing rule base. For example, if clustering reveals that the data source to be processed has known data source characteristics, the basic parsing rule base is incrementally updated based on the common parsing strategies corresponding to the known data source characteristics to obtain the target parsing rule base. If clustering reveals that the data source to be processed has unknown data source characteristics, the common parsing strategies corresponding to the known data source characteristics are used. A structural similarity projection algorithm is then used to heuristically infer the field mapping strategy of the data stream to be processed using protocol type, data format, and structural features, thereby dynamically generating the target parsing rule base. This embodiment does not specifically limit this process.

[0063] Here, by analyzing the structural similarity of historical parsing records, the evolutionary patterns of protocol structures can be automatically summarized from historical experience, and the content of the parsing rule base can be dynamically adjusted accordingly. This realizes the transformation of parsing rules from static preset to adaptive evolution, effectively reducing manual maintenance costs and improving the rule base's coverage of protocol variants.

[0064] For example, in one possible implementation, updating the basic parsing rule base based on the similarity to obtain the target parsing rule base includes: Aggregate target parsing records with a similarity greater than a preset clustering threshold from multiple historical parsing records into target parsing clusters; Extract the common parsing strategy among all target parsing records from the target parsing cluster; the common parsing strategy includes a common field mapping strategy and a common parsing path; Based on the shared parsing strategy, the basic parsing rule base is incrementally updated to obtain the target parsing rule base.

[0065] like Figure 2As shown, the clustering process of historical parsing records is scheduled at a preset period, which in this embodiment is preferably executed once a day. During each scheduling, target parsing records with a similarity value greater than a preset clustering threshold are aggregated into the same target parsing cluster, i.e., a regular cluster, based on the similarity value calculated by the structural similarity function. The preset clustering threshold here controls the minimum structural consistency required between historical parsing records grouped into the same cluster. The value of this threshold can be adjusted according to the accuracy requirements of the actual application scenario, such as 0.9, etc. This embodiment does not specifically limit this value.

[0066] After clustering, common parsing strategies among all target parsing records are extracted from the target parsing cluster. These common parsing strategies refer to the parsing logic that appears consistently across all historical parsing records within the same target parsing cluster. This includes two components: a common field mapping strategy, which is the consistent field name mapping and data type conversion rules used for the same semantic fields in each record; and a common parsing path, which is the consistent hierarchical traversal order and field extraction logic followed when unpacking data frames layer by layer in each record. The process of extracting common parsing strategies essentially involves performing an intersection operation on multiple parsing instances within the cluster, retaining the stable and unchanging core rule components, and filtering out occasional rule differences caused by the specificity of individual data sources.

[0067] Based on the extracted common parsing strategies, an incremental update operation is performed on the basic parsing rule base (also known as the rule template base). Incremental update refers to modifying or supplementing only the parts of the basic parsing rule base that differ from or are missing from the newly discovered common parsing strategies, without affecting existing, verified rule entries. Specifically, if a common parsing strategy contains a new rule that does not yet exist in the basic parsing rule base, the new rule is added to the base; if a rule in a common parsing strategy differs in parameters from a rule with the same name in the basic parsing rule base, the parameters are updated according to the new common parsing strategy. After the update, a target parsing rule base adapted to the current data stream is obtained.

[0068] The method provided in this embodiment combines periodic clustering and incremental updates, enabling the parsing rule base to continuously evolve without interrupting system operation. The clustering mechanism automatically merges parsing instances with similar structures, avoiding redundancy and expansion in the rule base; the incremental update mechanism ensures that the scope of each rule change is controllable and traceable, effectively balancing the evolution speed and stability of the rule base.

[0069] For example, in another possible embodiment, updating the basic parsing rule base based on the similarity to obtain the target parsing rule base further includes: When all similarities are less than or equal to the preset clustering threshold, the field mapping strategy of the data stream to be processed is inferred based on the protocol type, the data format, and the structural features. Generate an initial parsing rule template based on the field mapping strategy; Send the initial parsing rule template and the confidence label corresponding to the initial parsing rule template to the client; Receive the completion confirmation information of the initial parsing rule template returned by the client; Based on the completion confirmation information and the initial parsing rule template, the basic parsing rule base is incrementally updated to obtain the target parsing rule base.

[0070] like Figure 2 As shown, when the similarity between all historical parsed records is less than or equal to the preset clustering threshold, it indicates that the structural features of the current data stream to be processed do not have sufficiently similar historical references in the parsed record library. That is, the data stream belongs to the features of an unknown data source appearing for the first time or the features of a new data source. At this time, based on the protocol identification vector constructed in step 110, namely protocol type, data format, and structural features, the field mapping strategy of the data stream to be processed is quickly inferred through the structural similarity projection function. The structural similarity projection function here is used to project the protocol identification vector of the data stream to be processed onto the feature space of a known protocol structure, find the closest known protocol structure as a reference prototype, and heuristically infer the boundary position, data type, and semantic attribution of each field in the data stream to be processed, thereby generating a field mapping strategy.

[0071] Based on the inferred field mapping strategy, an initial parsing rule template (also known as the initial rule template) is automatically generated. This initial parsing rule template contains the inferred field boundary definitions, data type conversion rules, and basic parsing path logic. Since this template is generated entirely based on model inference and has not been manually verified, a low-confidence label is attached to it to indicate that the template's credibility is yet to be confirmed.

[0072] Furthermore, the initial parsing rule template and its corresponding confidence labels are sent to the client. Here, the client refers to the manual confirmation interface running on the terminal device used by operations personnel or domain experts. This interface provides a difference highlighting function, visually highlighting fields with low inference confidence in the initial parsing rule template; it also provides a field semantic hint function, offering candidate semantic suggestions for inferred fields based on similar entities in the knowledge graph; in addition, the interface provides an interactive editor, supporting manual correction or supplementation of field boundaries, data types, and semantic mappings, as well as information confirmation.

[0073] Once the client returns confirmation information for the initial parsing rule template, the template can be modified accordingly to obtain confirmed parsing rules. These confirmed rules are then incorporated into the basic parsing rule base for incremental updates, resulting in the target parsing rule base. Furthermore, these confirmed parsing rules can also be used as training samples, such as in optimizing the weight coefficients of the protocol similarity matching formula in step 110 and in the incremental training of the multi-dimensional data feature vector extraction model, thereby enhancing the system's ability to subsequently identify similar unknown protocols.

[0074] The method provided in this embodiment continuously expands the coverage of the rule base and maintains a steady increase in parsing accuracy through a mechanism that combines incremental learning and manual verification. This allows it to not only provide an initial parsing solution when facing entirely new data sources, but also to rapidly improve rule quality through human feedback. This mechanism effectively enhances the parsing versatility and evolutionary capability of new protocols, formats, and structures, and enables the parsing rule base to continuously expand its coverage and improve parsing accuracy while running continuously. This achieves an intelligent transformation from passive adaptation to proactive evolution in handling heterogeneous data sources.

[0075] Step 130: Parse the data stream to be processed according to the target parsing rule base to obtain the target structure data.

[0076] Optionally, after obtaining the target parsing rule base, the parsing rules defined in the target parsing rule base, such as field boundaries, offsets, field mapping relationships, data type conversion logic, and protocol conversion logic, can be used to perform frame-by-frame unpacking and format mapping operations on the data stream to be processed, thereby converting it into target structure data carried by a unified intermediate format.

[0077] The parsing here refers to the process of reconstructing heterogeneous raw bitstreams or message sequences into structured data with clear hierarchies and attribute definitions according to a preset logical topology. The target structured data not only retains the core content of the original data to ensure that the original semantics and format information are preserved during the conversion process, but also achieves semantic alignment of multi-source heterogeneous data within the system through standardized data containers. This embodiment effectively solves the problem of semantic loss caused by inconsistent formats in traditional parsing processes for heterogeneous data by constructing a unified intermediate format.

[0078] For example, in order to achieve structural compatibility and semantic consistency of multi-source heterogeneous data in subsequent processing stages, in one possible embodiment, the target structural data includes an original data layer, a metadata layer, and a semantic tag layer. The original data layer includes multiple components from the following: hierarchical structure information of the data stream to be processed, binary and text features, data packet boundary information, and frame positioning information. The metadata layer includes metadata description information organized in a key-value structure and a data quality score; the data quality score is obtained by weighted summation of field parsing completeness, field semantic matching confidence, and field value verification consistency; the metadata description information includes data source identifier, collection timestamp, protocol version number, and a field definition dictionary used to identify parsing result attributes; The semantic tagging layer includes structured semantic tags; the structured semantic tags are generated by using a multi-tag classifier to perform entity recognition, concept classification, and semantic relationship annotation on the field values ​​of each field in the data stream to be processed, based on the structural docking relationship between the metadata layer and the knowledge graph.

[0079] Figure 3 This is a schematic diagram of the hierarchical structure of the target structure data provided by the present invention.

[0080] like Figure 3 As shown, a three-layer structure encapsulation strategy can be adopted to encapsulate the parsing results of the data stream to be processed into target structured data. The target structured data here includes a raw data layer, a metadata layer, and a semantic tag layer. Each layer, while maintaining the integrity of the data content, undertakes different levels of semantic mapping and context enhancement.

[0081] Specifically, the raw data layer includes several elements from the hierarchical structure information (structure mapping tree) of the data stream to be processed, binary and text features, packet boundary information, and frame location information. The raw data layer preserves the hierarchical structure of the raw data stream, its dual binary and text representations, and its packet boundary and frame location information within the protocol through a structure mapping tree. This layer is a complete reproduction of the identification and parsing output results from steps 110 and 120, ensuring that all subsequent processing can be based on this data for backtracking and reconstruction.

[0082] The metadata layer includes key-value structured metadata descriptions and key attributes such as data quality scores. The metadata descriptions include the data source identifier (also known as the data source ID), collection timestamp, protocol version number, and a field definition dictionary to identify the attributes of the parsed results. Through structured attribute descriptions, the metadata layer provides a unified identity and self-explanatory capability for heterogeneous data. The field definition dictionary is used to identify the name, data type, value range, and semantic description of each field in the parsed results in a structured manner.

[0083] The data quality score here can be calculated using the following formula: ; Among them, Represents a data unit in the data stream to be processed. Data quality score, Representing data unit The field parsing completeness of the j-th field is used to measure whether the field has been accurately parsed and mapped. Representing data unit The semantic matching confidence score of the j-th field can be generated by the knowledge graph entity alignment function. Representing data unit The consistency of the j-th field value is evaluated based on the deviation rate between the field constraint rules and the actual value; α, β, and γ are weighting coefficients, which can be adjusted according to the protocol type and application scenario to ensure that the scoring system has cross-protocol data consistency and comparability; m is the data unit. The scoring system comprehensively quantifies the overall quality of data units in terms of parsing completeness, semantic accuracy, and value range compliance through a weighted fusion of three orthogonal dimensions, ensuring consistency and comparability in quality evaluation across cross-protocol data.

[0084] The semantic tagging layer is built on top of the metadata layer and includes structured semantic tags. During the acquisition of structured semantic tags, a multi-tag classifier based on graph path embedding can be used. According to the structural connection between the metadata layer and the pre-built domain knowledge graph, three operations are performed on the field values ​​of each field in the data stream to be processed: entity recognition, i.e., determining the specific entity instance corresponding to the field value in the knowledge graph; concept classification, i.e., classifying the entity to which the field value belongs into the concept hierarchy category in the knowledge graph; and semantic relationship annotation, i.e., annotating the semantic relationship type between the field value and other field values ​​according to the relationship definition between entities in the knowledge graph. The results of these three operations together constitute the structured semantic tags, forming a structured semantic graph with data fields as nodes and semantic relationships as edges.

[0085] The method provided in this embodiment constructs a unified intermediate format that plays a core bridging role throughout the entire process through a three-layer encapsulation design of the raw data layer, metadata layer, and semantic tag layer. This target structured data can not only receive the physical results of front-end protocol identification and parsing, but also provide a unified semantic foundation for back-end data fusion, modeling analysis, and visualization. Its structured design ensures full lifecycle traceability, field-level tracking, and cross-source semantic consistency of data, effectively supporting the generalization and deployment capabilities of the underlying acquisition and parsing system in complex industrial scenarios and multiple business domains, and significantly improving the quality of heterogeneous data access and its subsequent utilization value.

[0086] Step 140: Perform semantic enhancement and consistency verification on the target structure data, and output standardized parsed data of the data stream to be processed based on the enhanced target structure data and the consistency verification results.

[0087] Optionally, after obtaining the target structure data, deep processing can be performed on it to ensure data compatibility and parsability.

[0088] Specifically, this can involve semantic enhancement and consistency verification of the target structure data. Semantic enhancement refers to using domain knowledge graphs to perform entity recognition and association modeling on the parsed fields, uncovering implicit data relationships hidden behind physical fields, and transforming the data from a single syntactic fragment into rich knowledge nodes. Consistency verification, on the other hand, uses a rule engine to strictly control the logical completeness of the data, ensuring that field values, data types, and protocol logic meet preset compliance constraints.

[0089] For example, in one possible implementation, the semantic enhancement of the target structural data includes: A heterogeneous attribute graph is constructed using the metadata description information and the structured semantic tags as graph nodes; Based on the weight coefficients corresponding to each edge type and the attention coefficients between each graph node, the representation vectors of each graph node in the heterogeneous attribute graph are updated, and the implicit associations between each field are obtained based on the update results. Based on the implicit association, semantic enhancement is performed on the target structure data to obtain the enhanced target structure data.

[0090] Optionally, when semantically enhancing the target structured data, a graph neural network model can be introduced to perform structured graph modeling and semantic propagation on the metadata layer in the target structured data.

[0091] Specifically, a heterogeneous attribute graph is constructed by using data source identifiers, field definition dictionaries, protocol version numbers, and structured semantic tags from the semantic tag layer as graph nodes. In this heterogeneous attribute graph, the graph nodes are connected by edges, and the edge weights are used to represent the co-occurrence frequency, semantic similarity, and temporal correlation between fields, thus forming a heterogeneous attribute graph that can fully depict the multidimensional relationships between fields.

[0092] The heterogeneous attribute graph here refers to a graph structure containing multiple types of nodes and edges. Using the components of metadata description information and structured semantic tags as graph nodes, it can simultaneously accommodate physical-level protocol attribute nodes and semantic-level tag nodes within the same graph, thus providing a structural foundation for subsequent semantic propagation. The co-occurrence frequency in the edge weights reflects the statistical strength of the co-occurrence of two fields in historical parsing records; semantic similarity measures the conceptual closeness of two fields in the knowledge graph space; and temporal correlation characterizes the synchronous or causal relationship between two fields in the time series of data collection. By simultaneously encoding these three association dimensions as edge weights, the heterogeneous attribute graph can comprehensively characterize the association strength between fields from three orthogonal dimensions: statistical, semantic, and temporal.

[0093] After constructing the heterogeneous attribute graph, a heterogeneous graph attention mechanism based on edge type weighting is used to iteratively update the representation vectors of each node in the graph. The core propagation function of this mechanism is defined as follows: ; in, Indicates that node v is at the th The representation vector after layer iteration. Let v represent the set of nodes adjacent to node v under edge type t. It is the set of all edge types in the heterogeneous attribute graph; , where is the weight coefficient corresponding to edge type t, used to map the features of neighbor nodes under different edge types to a unified semantic space; σ represents the attention coefficient between node u and node v. This attention coefficient is dynamically calculated by combining the similarity between semantic label pairs and historical co-occurrence frequency. The larger the value, the more significant the information transmission contribution of node u to node v. σ is a non-linear activation function. This propagation mechanism gradually strengthens the semantic coupling between key fields during training, enabling the system to recognize weakly related field relationships. It is particularly suitable for scenarios where field names are inconsistent but semantics are consistent in multi-source data fusion, significantly improving the performance of intermediate formats in data connectivity, knowledge completion, and semantic unification.

[0094] In the aforementioned propagation process, each iteration aggregates the semantic information of adjacent nodes into the current node after attention weighting. After multiple iterations, the node's representation vector not only encodes its own attribute features but also incorporates its semantic association information within the multi-hop neighborhood of the graph structure. Based on the updated node representation vectors, the vector distance or similarity between node pairs can be calculated to identify implicit associations that were not explicitly related at the physical field level but were highly coupled in the semantic space.

[0095] Implicit relationships here refer to potential dependencies or equivalences between fields that are not explicitly declared in the original data through field references or foreign key constraints, but are revealed after semantic propagation by a graph neural network. This implicit relationship is particularly suitable for scenarios involving multi-source data fusion where field names are inconsistent but semantically consistent. For example, data streams from devices from different manufacturers may use "temp" and "temperature" as the names of the temperature field, which are completely different in physical messages. However, after semantic propagation by a graph neural network, their node representation vectors will be highly similar, allowing the system to automatically establish a semantic equivalence relationship between them.

[0096] Subsequently, based on the implicit relationships identified above, association completion and semantic enhancement operations can be performed on the semantic label layer in the target structure data, including supplementing missing semantic association edges, correcting semantic labels with weak confidence, and establishing unified semantic mapping nodes for newly discovered cross-source equivalent fields, thereby obtaining the enhanced target structure data.

[0097] The method provided in this embodiment, by introducing the heterogeneous graph attention mechanism of graph neural networks, can not only handle explicitly declared data relationships, but also actively mine deep semantic associations hidden between heterogeneous data sources. This significantly improves the expressiveness of target structured data in terms of data connectivity, knowledge completion, and semantic unity, providing high-quality data input with richer semantics and more complete associations for data fusion and analysis.

[0098] For example, to ensure that the semantically enhanced target structured data meets the usage requirements of downstream systems in terms of structural integrity and semantic consistency, in one possible implementation, the consistency verification steps include: A rule-based validation engine is used to rigorously verify the structural integrity and semantic consistency of the enhanced target structured data. This consistency verification is a dual-checking mechanism oriented towards both structural specifications and semantic logic, aiming to ensure that the parsed and enhanced data conforms to both the structural constraints of the target format and the semantic logic requirements of the business domain.

[0099] Specifically, the validation engine first automatically selects the corresponding standard rule set for hierarchical constraint validation based on the data format of the enhanced target structure data. For example, for data in the eXtensible Markup Language (XML) structure, the XML Path Language (XPath) rule set is used for hierarchical constraint validation; for data in the JavaScript Object Notation (JSON) structure, standard rule sets such as XPath and JSONSchema are used for structure validation. The purpose of hierarchical constraint validation is to confirm whether the nesting level, number of nodes, and parent-child relationships between nodes conform to the structural template defined by the protocol specification.

[0100] In addition, the validation engine can perform field-level logical validation. This field-level logical validation includes at least several of the following dimensions: Field value type validation: Verify whether the data type of the actual value of each field is consistent with the data type declared in the field definition dictionary; Required field integrity check: Verify that all required fields defined in the protocol specification corresponding to the current protocol version have been successfully parsed and are not empty, ensuring that critical business information is not missing; Semantic tag compatibility verification: Verify whether the structured semantic tags in the semantic tag layer match the field semantics defined in the current protocol version, and avoid semantic tagging errors caused by protocol version iteration.

[0101] For any data unit that fails any of the above checks, the verification engine automatically marks it as pending and writes the corresponding violation item, violation field, and violation reason into the consistency verification result. Simultaneously, the verification engine determines whether to trigger an event alarm based on whether the decrease in the data quality score of the data unit before and after verification exceeds a preset confidence level decrease threshold. When the decrease exceeds the confidence level decrease threshold, an alarm event is automatically generated, and the alarm information is sent to the system logs, the manual review interface, or the upstream platform so that operations and maintenance personnel can intervene promptly.

[0102] The confidence level drop threshold here refers to a preset numerical threshold used to determine whether the decline in data quality score has reached a level that requires an alert. Specifically, it can be configured according to the tolerance of data quality in the specific application scenario.

[0103] Subsequently, based on the enhanced target structure data and the consistency verification results, standardized parsed data of the data stream to be processed is output. Here, standardized parsed data refers to the final output data that has undergone a full process of protocol identification, rule parsing, three-layer structure encapsulation, semantic enhancement, and consistency verification, possessing complete structural information, high-quality metadata description, rich semantic annotation, and trustworthiness markers. Data units that pass consistency verification are directly output as highly reliable standardized parsed data to the target system or storage layer; data units marked as pending processing are output along with the violation information from their consistency verification results, allowing downstream systems to choose whether to accept, downgrade, or wait for manual correction and re-injection based on business needs.

[0104] The output method here can be to write the standardized parsed data into a distributed storage system, push it to a message queue, or return it directly to the requester through an application programming interface (API) to support subsequent data analysis, fusion processing, or visualization.

[0105] Here, through a dual mechanism of hierarchical constraint verification and field-level logical validation, high-quality integrated structural and semantic data input is provided to downstream analysis systems while ensuring structural security and semantic accuracy. This ensures that the underlying data acquisition and parsing system is capable of being deployed in application scenarios with extremely high data consistency requirements, such as industrial scenarios, medical data, and financial logs. Simultaneously, an alarm triggering mechanism based on confidence level reduction can promptly detect and isolate abnormal data, effectively preventing low-quality data from contaminating downstream decision-making systems and providing reliable data quality assurance for application scenarios with high consistency requirements.

[0106] The method provided in this embodiment achieves end-to-end automated conversion from raw heterogeneous data streams to high-quality standardized parsed data by performing multi-dimensional data feature vector extraction and protocol attribute identification, adaptive parsing rule base construction, three-layer unified intermediate format conversion, semantic enhancement and consistency verification on the data stream to be processed. It effectively solves the problems of insufficient adaptability, loss of semantic information and difficulty in guaranteeing data quality in multi-protocol environments of traditional parsing technology, and significantly improves the intelligence level and processing efficiency of multi-source heterogeneous data access.

[0107] Based on the above embodiments, in order to fully utilize existing metadata information to achieve intelligent dynamic allocation of resources during the output stage of standardized parsed data, thereby ensuring the timeliness of high-value data stream output, in one possible implementation, the output step of the standardized parsed data includes: The priority coefficient of the data stream to be processed is determined based on the data source identifier, the collection timestamp, the protocol version number, and the data quality score. Based on the priority coefficient, and the queuing time, actual processing time and maximum tolerable latency of the data stream to be processed under each candidate resource scheduling strategy, the target resource scheduling strategy is determined among the multiple candidate resource scheduling strategies. According to the target resource scheduling strategy, the parsing engine performs a resource reallocation action, and using the parsing engine after resource reallocation, based on the enhanced target structure data and the consistency verification result, outputs the standardized parsed data of the data stream to be processed to the data receiving end.

[0108] Optionally, the underlying data acquisition and parsing system also includes a dynamic resource scheduling module. This module, acting as the system's control center, runs throughout the entire data parsing and output process. While ensuring high throughput and low latency processing capabilities, it dynamically adjusts the parsing engine's resource allocation strategy based on real-time data stream status and priority tags. Unlike traditional systems where resource scheduling is an operation separate from the data processing pipeline, this step embeds resource scheduling decisions into the output stage of standardized parsed data. This ensures that each data stream, after completing semantic enhancement and consistency verification, receives sufficient computing resources commensurate with its business value before entering the final delivery stage. This design avoids delivery delays caused by resource bottlenecks in the output stage for data that has already undergone high-quality processing in the parsing stage, ensuring that the entire chain from data access to final output is under intelligent management.

[0109] Figure 4 This is a schematic diagram of the resource scheduling process provided by the present invention; for example... Figure 4 As shown, during resource scheduling, four attributes—data source identifier, collection timestamp, protocol version number, and data quality score—can be extracted from the metadata layer of the target structure data. Based on these attributes, the priority coefficient (also known as the priority index) of the data stream to be processed can be calculated.

[0110] The priority coefficient here is a comprehensive quantitative indicator used to characterize the urgency with which the data stream to be processed and output is prioritized in the current system operating environment. This priority coefficient is not determined by a single attribute, but rather by a comprehensive calculation of the four attributes mentioned above. The contribution logic of each attribute to the priority coefficient is as follows: Data source identifiers are used to distinguish different data-generating devices or business system instances. In actual deployments, the business importance of different data sources varies. The system pre-configures a corresponding business importance level for each data source identifier, and this level serves as one of the fundamental components for calculating the priority coefficient. Data collection timestamps are used to measure the timeliness of data. Data streams with timestamps closer to the current moment have higher value in supporting real-time decision-making and are therefore given a higher weight in terms of timeliness. Conversely, data streams collected a long time ago have a lower weight in terms of timeliness. By introducing collection timestamps as a priority calculation factor, when multiple data streams are queued concurrently, data streams with higher timeliness requirements can be prioritized and output as quickly as possible.

[0111] Protocol version numbers reflect the protocol complexity of a data stream. Newer versions typically contain richer field definitions and more complex nested structures, requiring more computational resources for parsing and output. Therefore, based on the protocol complexity assessment value corresponding to the protocol version number, a suitable priority boost is applied to high-complexity protocol data streams in the priority coefficient to ensure that such data streams are not continuously delayed due to longer processing times when resources are scarce.

[0112] Data quality scores directly reflect the overall quality level of a data stream after parsing, encapsulation, and enhancement. Data streams with higher quality scores are more valuable to downstream analysis systems and therefore receive higher weight in priority calculations. Conversely, data streams with lower quality scores, especially those marked as pending in consistency verification, have lower priority coefficients, allowing the system to allocate limited output resources preferentially to high-quality data.

[0113] In practice, the four attributes mentioned above can be quantified and normalized, and then a final priority coefficient for the data stream to be processed can be generated through weighted summation or other methods. The larger the priority coefficient, the higher the resource acquisition priority of the data stream in subsequent resource scheduling stages.

[0114] The resource dynamic scheduling module maintains a data flow status table in real time. This status table records information such as the priority coefficient, current queuing position, amount of resources consumed, and estimated remaining processing time for each active data flow in the current system. It also continuously integrates traffic fluctuation trends with task completion time windows to provide real-time data support for subsequent scheduling decisions.

[0115] After obtaining the priority coefficients of the data stream to be processed, the resource scheduling strategy selection phase begins. Here, candidate resource scheduling strategies refer to multiple predefined or dynamically generated resource allocation schemes. Each candidate resource scheduling strategy corresponds to a specific computing resource allocation method, including but not limited to the specific combination of parameters such as the method and number of parsing threads allocated to the data stream, the number of CPU cores, memory quota, and network output bandwidth.

[0116] For each candidate resource scheduling strategy, based on the data flow status table and by integrating traffic fluctuation trends and task completion time windows, evaluate three time metrics of the data flow to be processed under that strategy: Queuing time refers to the estimated waiting time for the data stream to be processed from entering the output queue to the start of the parsing engine executing the output operation under the candidate resource scheduling strategy.

[0117] Actual processing time refers to the estimated time required for the parsing engine to complete the output of all standardized parsed data under the candidate resource scheduling strategy, from the start of output operations. This time depends on the size of the data stream to be processed, the protocol complexity, and the amount of computing resources allocated by the strategy.

[0118] Maximum tolerable latency refers to the maximum time limit allowed from the time the data enters the system to the time the final output is completed, as set in the service level agreement corresponding to the data stream to be processed.

[0119] Subsequently, based on the priority coefficients and the three time metrics mentioned above, the strategy that minimizes the weighted performance loss is selected as the target resource scheduling strategy from among multiple candidate resource scheduling strategies using the following objective function: ; in, This represents the weighted performance loss function under candidate resource scheduling strategy θ. and Let be the queuing time and the actual processing time of the i-th data stream under the candidate resource scheduling policy θ, respectively. The maximum tolerable latency set in the service level agreement for the i-th data stream. The priority weight of the i-th data stream is the quantized weight calculated in the previous steps based on the data source identifier, collection timestamp, protocol version number, and data quality score. This represents the total number of active data streams in the current system. This objective function drives the scheduling controller to make resource reallocation decisions with the goal of minimizing the latency of high-priority streams, prioritizing the data path of core data streams when computational resources are limited.

[0120] When multiple candidate resource scheduling strategies have the same weighted performance loss function value, the strategy with the lowest total resource consumption will be selected as the target resource scheduling strategy to maximize resource utilization efficiency while ensuring output timeliness and avoid unnecessary resource waste.

[0121] Therefore, by incorporating priority coefficients and time constraints into a unified objective function, this scheduling decision mechanism upgrades from the traditional static allocation or first-come-first-served model to a business value-oriented intelligent dynamic scheduling model, enabling resource allocation to be optimized in real time based on the current system load status and the business attributes of each data stream.

[0122] Once the target resource scheduling policy is obtained, the parsing engine can be reassigned accordingly. This resource reassignment refers to the process by which the dynamic resource scheduling module reallocates the computing resources currently held by the parsing engine according to the thread-level and instance-level scheduling instructions contained in the target resource scheduling policy.

[0123] In practical execution, a lightweight runtime unit for the parsing engine can be built based on containerization technology. Utilizing the Horizontal Pod Autoscaler mechanism provided by the Kubernetes cluster, the number of replicas is adjusted according to the concurrent processing capacity requirements determined in the instance-level scheduling instructions. When the target resource scheduling policy determines that the current system requires higher concurrent processing throughput, the number of container instance replicas of the parsing engine can be automatically increased based on runtime metrics such as CPU utilization, memory usage, and parsing duration, achieving horizontal elastic scaling of computing resources. When the target resource scheduling policy determines that the current concurrent pressure has fallen back to normal levels, the number of replicas is automatically reduced to release excess resources and avoid unnecessary resource consumption. Furthermore, high-level data streams are bound to independent container threads using thread-level scheduling instructions, and the parsing process is ensured stable computing resources and bandwidth through affinity policies and resource QoS restrictions. Simultaneously, the resource dynamic scheduling module maintains a parsing thread pool and container resource mapping table internally, recording in real time the binding relationship between each parsing thread and the container instance it runs on, as well as the current CPU utilization, memory usage, network throughput, and task queue depth of each container instance, among other runtime status information. When the system detects a dynamic surge in data flow, such as an industrial data source generating significantly more data than normal in a short period due to batch device startup, or when the processing time of a parsing thread approaches a bottleneck threshold (i.e., the actual processing time is close to a preset percentage threshold of the maximum tolerable latency), the scheduling controller automatically launches temporary container nodes from the resource pool to replenish resources. These temporary container nodes are immediately added to the parsing thread pool after creation, sharing the previously backlogged parsing and output tasks. This achieves millisecond-level horizontal scaling and parsing latency compression, effectively supporting the system's continuous operational stability and elastic load handling capabilities in complex protocol scenarios, high-concurrency data paths, and edge computing node deployments. Once the surge in traffic returns to normal levels and the temporary container node's task queue is cleared, the resource dynamic scheduling module automatically reclaims the temporary container node and releases its occupied resources, ensuring that the system resource pool does not experience long-term redundant occupation due to temporary expansion.

[0124] After both the thread-level scheduling instructions and instance-level scheduling instructions have been executed, the parsing engine after resource reallocation performs the final output operation of standardized parsing data based on the enhanced target structure data and consistency verification results.

[0125] The data receiving end here refers to the downstream system or storage component that receives standardized parsed data, including but not limited to the target system, such as the business analysis application system, or the storage layer.

[0126] The dynamic resource scheduling module, in conjunction with the dynamic protocol identification, adaptive rule generation, and unified intermediate format conversion processes, ensures the real-time performance, controllability, and scalability of the entire parsing chain. This module not only operates in the standardized parsing data output stage but also allocates resources upfront based on the real-time computing load of the protocol identification and rule generation modules. This ensures the system maintains stable processing power and timely response speed even when facing scenarios such as concurrent access of multi-source heterogeneous data, complex protocol mixed parsing, and peak traffic surges.

[0127] The method provided in this embodiment embeds three stages—priority quantization, objective function-driven strategy selection, and containerized elastic resource reallocation—into the output stage of standardized parsing data. This enables thread-level scheduling instructions to provide resource isolation guarantees for independent parsing threads for high-priority data streams, while instance-level scheduling instructions achieve on-demand scaling of parsing engine computing resources through containerization technology. The collaborative execution of these two types of instructions effectively avoids resource bottlenecks and indiscriminate queuing problems in the output stage, ensuring the timeliness of critical data stream delivery and the operational stability of the system in high-concurrency environments.

[0128] In summary, the data processing method and system provided in this application have the following comprehensive advantages: At the protocol identification level, multi-dimensional data feature vector extraction based on joint modeling of word frequency-inverse document frequency weight distribution function and character-level N-gram statistical model, combined with normalized weighted similarity matching mechanism, achieves high-accuracy dynamic identification and zero-configuration automatic adaptation of protocol types for multi-source heterogeneous data streams, significantly reducing the need for manual intervention, and ensuring the continuous expansion capability of the protocol library through the fallback mechanism of manual intervention interface and custom rule configuration channel.

[0129] At the rule generation level, a clustering and inductive method based on incremental learning is used to perform periodic structural similarity analysis on historical parsing records. Combined with structural similarity projection for unknown data sources and a manual verification feedback mechanism, the parsing rule base is transformed from static preset to adaptive self-evolution. This enables the system to quickly respond to the diversity and changes of emerging data sources, continuously expand the rule coverage, and maintain the growth of parsing accuracy.

[0130] At the data structure and semantic level, a three-layer unified intermediate format encapsulation design consisting of a raw data layer, a metadata layer, and a semantic tag layer, combined with a multi-label classification and confidence quantification scoring system based on domain knowledge graphs, achieves structural compatibility and semantic fusion of cross-protocol data. While preserving the integrity and traceability of the original data, it endows the data with deep semantic meaning, greatly enhancing the basic support capabilities for subsequent data analysis, fusion, and visualization.

[0131] In terms of data quality assurance, the implicit relationships between fields are mined through the semantic propagation of graph neural networks based on the heterogeneous graph attention mechanism with edge type weighting. Combined with the dual rule engine mechanism of hierarchical constraint verification and field-level logical verification, as well as the anomaly alarm triggering based on confidence reduction, the integrity and consistency of the data are ensured, and the semantic understanding depth and intelligent discrimination level of the system are improved.

[0132] At the resource scheduling level, by using multi-dimensional priority quantization based on metadata attributes, scheduling strategy selection driven by response time optimization objective function, and independent thread allocation and elastic replica scaling based on containerization technology, the system's stability and real-time response capability under high load and multi-task concurrency environments are effectively guaranteed, achieving the optimal balance between computing resource utilization efficiency and the timeliness of critical data stream delivery.

[0133] Overall, the method provided in this application constructs a complete intelligent data parsing link from protocol identification, rule generation, format conversion, semantic enhancement, quality verification to resource scheduling. The data flow of each link is closely connected and increases value at each level, which significantly improves the intelligence level and processing efficiency of multi-source heterogeneous data access, and provides a solid data foundation and technical guarantee for multi-domain big data integration and intelligent analysis.

[0134] The data processing system provided by the present invention is described below. The data processing system described below and the data processing method described above can be referred to in correspondence.

[0135] Figure 5 This is a schematic diagram of the data processing system provided by the present invention; as shown below. Figure 5 As shown, the system includes: The dynamic protocol identification and adaptation module 510 is used to obtain the protocol type, data format and structural features of the data stream to be processed based on the multi-dimensional data feature vector of the data stream to be processed; The adaptive parsing rule generation module 520 is used to construct a target parsing rule library that matches the data stream to be processed based on the protocol type, the data format, and the structural features. The unified intermediate format conversion module 530 is used to parse the data stream to be processed according to the target parsing rule library to obtain the target structure data; Metadata enhancement and verification module 540 is used to perform semantic enhancement and consistency verification on the target structure data; The standardized output module 550 is used to output standardized parsed data of the data stream to be processed based on the enhanced target structure data and the consistency verification results.

[0136] In addition, the system also includes a resource dynamic scheduling module 560 for dynamically allocating resources to the dynamic protocol identification and adaptation module, the adaptive parsing rule generation module, the unified intermediate format conversion module, the metadata enhancement and verification module, and the standardized output module.

[0137] The system provided in this embodiment achieves end-to-end automated conversion from raw heterogeneous data streams to high-quality standardized parsed data through end-to-end processing, including multi-dimensional data feature vector extraction and protocol attribute identification of the data stream to be processed, adaptive parsing rule base construction, three-layer unified intermediate format conversion, semantic enhancement and consistency verification. It effectively solves the problems of insufficient adaptability, loss of semantic information and difficulty in guaranteeing data quality in multi-protocol environments of traditional parsing technologies, and significantly improves the intelligence level and processing efficiency of multi-source heterogeneous data access.

[0138] The system provided by this invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0139] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a data processing method, which includes: obtaining the protocol type, data format, and structural features of the data stream to be processed based on its multidimensional data feature vector; constructing a target parsing rule base matching the data stream to be processed based on the protocol type, data format, and structural features; parsing the data stream to be processed according to the target parsing rule base to obtain target structural data; performing semantic enhancement and consistency verification on the target structural data; and outputting standardized parsed data of the data stream to be processed based on the enhanced target structural data and the consistency verification results.

[0140] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the data processing methods provided by the above methods. The method includes: obtaining the protocol type, data format, and structural features of the data stream to be processed based on the multidimensional data feature vector of the data stream to be processed; constructing a target parsing rule base matching the data stream to be processed based on the protocol type, the data format, and the structural features; parsing the data stream to be processed based on the target parsing rule base to obtain target structural data; performing semantic enhancement and consistency verification on the target structural data; and outputting standardized parsed data of the data stream to be processed based on the enhanced target structural data and the consistency verification results.

[0142] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the data processing methods provided by the above methods. The method includes: obtaining the protocol type, data format, and structural features of the data stream to be processed based on a multidimensional data feature vector; constructing a target parsing rule base matching the data stream to be processed based on the protocol type, the data format, and the structural features; parsing the data stream to be processed according to the target parsing rule base to obtain target structure data; performing semantic enhancement and consistency verification on the target structure data; and outputting standardized parsed data of the data stream to be processed based on the enhanced target structure data and the consistency verification results.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized by, include: Based on the multidimensional data feature vector of the data stream to be processed, the protocol type, data format, and structural features of the data stream to be processed are obtained; Based on the protocol type, the data format, and the structural features, construct a target parsing rule base that matches the data stream to be processed; The target parsing rule base is used to parse the data stream to be processed to obtain the target structure data; The target structure data is semantically enhanced and consistency verified. Based on the enhanced target structure data and the consistency verification results, the standardized parsed data of the data stream to be processed is output.

2. The data processing method according to claim 1, characterized in that, The step of constructing a target parsing rule base that matches the data stream to be processed based on the protocol type, the data format, and the structural features includes: Based on the protocol type, the structural features, and the data format, load multiple historical parsing records that match the data stream to be processed into the parsing record library; Calculate the similarity of each of the historical parsed records in the field structure feature space; Based on the similarity, the basic parsing rule base is updated to obtain the target parsing rule base.

3. The data processing method according to claim 2, characterized in that, The step of updating the basic parsing rule base based on the similarity to obtain the target parsing rule base includes: Aggregate target parsing records with a similarity greater than a preset clustering threshold from multiple historical parsing records into a target parsing cluster; Extract the common parsing strategy among all target parsing records from the target parsing cluster; the common parsing strategy includes a common field mapping strategy and a common parsing path; Based on the shared parsing strategy, the basic parsing rule base is incrementally updated to obtain the target parsing rule base.

4. The data processing method according to claim 3, characterized in that, The method further includes: When all similarities are less than or equal to the preset clustering threshold, the field mapping strategy of the data stream to be processed is inferred based on the protocol type, the data format, and the structural features. Generate an initial parsing rule template based on the field mapping strategy; Send the initial parsing rule template and the confidence label corresponding to the initial parsing rule template to the client; Receive the completion confirmation information of the initial parsing rule template returned by the client; Based on the completion confirmation information and the initial parsing rule template, the basic parsing rule base is incrementally updated to obtain the target parsing rule base.

5. The data processing method according to any one of claims 1-4, characterized in that, The target structure data includes a raw data layer, a metadata layer, and a semantic tag layer; The original data layer includes multiple components from the following: the hierarchical structure information of the data stream to be processed, binary and text features, data packet boundary information, and frame positioning information. The metadata layer includes metadata description information organized in a key-value structure and a data quality score; the data quality score is obtained by weighted summation of field parsing completeness, field semantic matching confidence, and field value verification consistency; the metadata description information includes data source identifier, collection timestamp, protocol version number, and a field definition dictionary used to identify parsing result attributes; The semantic tagging layer includes structured semantic tags; the structured semantic tags are generated by using a multi-tag classifier to perform entity recognition, concept classification, and semantic relationship annotation on the field values ​​of each field in the data stream to be processed, based on the structural docking relationship between the metadata layer and the knowledge graph.

6. The data processing method according to claim 5, characterized in that, The semantic enhancement of the target structure data includes: A heterogeneous attribute graph is constructed using the metadata description information and the structured semantic tags as graph nodes; Based on the weight coefficients corresponding to each edge type and the attention coefficients between each graph node, the representation vectors of each graph node in the heterogeneous attribute graph are updated, and the implicit associations between each field are obtained based on the update results. Based on the implicit association, semantic enhancement is performed on the target structure data to obtain the enhanced target structure data.

7. The data processing method according to claim 5, characterized in that, The steps for outputting the standardized parsed data include: The priority coefficient of the data stream to be processed is determined based on the data source identifier, the collection timestamp, the protocol version number, and the data quality score. Based on the priority coefficient, and the queuing time, actual processing time and maximum tolerable latency of the data stream to be processed under each candidate resource scheduling strategy, the target resource scheduling strategy is determined among the multiple candidate resource scheduling strategies. According to the target resource scheduling strategy, the parsing engine performs a resource reallocation action, and using the parsing engine after resource reallocation, based on the enhanced target structure data and the consistency verification result, outputs the standardized parsed data of the data stream to be processed to the data receiving end.

8. The data processing method according to any one of claims 1 to 4, characterized in that, The step of obtaining the protocol type, data format, and structural features of the data stream to be processed based on its multidimensional data feature vector includes: For each dimension of data feature vector, calculate the dimensional feature similarity between the data stream to be processed and each candidate protocol parsing template in the parsing template library; Based on the weight coefficients corresponding to the feature similarity of each dimension, the feature similarity of each dimension is weighted and summed, and then normalized to obtain the matching score between the data stream to be processed and each of the candidate protocol parsing templates. Among the multiple candidate protocol parsing templates, the candidate protocol parsing templates with matching scores greater than a preset score threshold are selected. If there are multiple candidate protocol parsing templates, then the target protocol parsing template with the highest matching score is selected from the multiple candidate protocol parsing templates; Extract the protocol type, data format, and structural features of the data stream to be processed from the target protocol parsing template.

9. The data processing method according to any one of claims 1 to 4, characterized in that, The steps for obtaining the multidimensional data feature vector of the data stream to be processed include: Multiple data packets are extracted from the data stream to be processed, and the header information of the multiple data packets is aggregated by window sliding to obtain aggregated header information; Based on the aggregated header information, character sequence statistical features and field weight distribution features are extracted from the data stream to be processed to obtain a multidimensional data feature vector of the data stream to be processed; the multidimensional data feature vector includes multiple features such as field length distribution features, separator frequency features, field pattern overlap features, and control character position features.

10. A data processing system, characterized by include: The dynamic protocol identification and adaptation module is used to obtain the protocol type, data format and structural features of the data stream to be processed based on the multi-dimensional data feature vector of the data stream to be processed; An adaptive parsing rule generation module is used to construct a target parsing rule library that matches the data stream to be processed based on the protocol type, the data format, and the structural features. A unified intermediate format conversion module is used to parse the data stream to be processed according to the target parsing rule base to obtain target structure data; The metadata enhancement and verification module is used to perform semantic enhancement and consistency verification on the target structure data; The standardized output module is used to output standardized parsed data of the data stream to be processed based on the enhanced target structure data and the consistency verification results.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1 to 9.

Images