A city-level intelligent community multi-source heterogeneous data standardization management method

Abstract

The application relates to the technical field of data processing, and discloses a city-level intelligent community multi-source heterogeneous data standardization management method, which comprises the following steps: acquiring heterogeneous data messages and generating a structure topology fingerprint, performing differential processing according to a fingerprint comparison result; when the comparison result is inconsistent, extracting time sequence statistical features of unknown fields to construct a physical feature vector, and calculating the topological structure correlation among the unknown fields to construct a topological distance matrix; based on the physical feature vector and a standard field reference vector, combining the topological distance matrix to construct and solve a global mapping loss function, and establishing the mapping relationship between the unknown fields and the standard fields; generating a streaming mask according to the mapping relationship and a message network communication protocol type, and extracting and storing standardized data based on the mask. The application can perceive message structure changes through a structure fingerprint, and can constrain a mapping relationship by combining space-time joint features, so that the accuracy of heterogeneous data mapping conversion and the system processing efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a standardized governance method for multi-source heterogeneous data in city-level smart communities. Background Technology

[0002] With the advancement of smart community construction in cities, the number of IoT sensing devices connected to the system platform is rapidly increasing. These devices are provided by different manufacturers and use different network communication protocols and underlying message encoding mechanisms, resulting in massive amounts of multi-source heterogeneous data. In order to achieve centralized management and collaborative analysis of various business scenarios at the platform level, it is necessary to convert this heterogeneous data with messy sources and different structures into unified standard structured data for storage.

[0003] Existing methods for standardizing multi-source heterogeneous data typically rely on manually pre-written data parsing templates and field mapping rules. In real-world operating environments, firmware upgrades of underlying devices, network protocol adjustments, or changes in business requirements often lead to unpredictable changes in the internal structure of data packets. Parsing methods based on static, manually defined rules cannot automatically detect and adapt to these structural changes, often requiring maintenance personnel to re-intervene and modify configuration code. This results in high system maintenance costs and fails to meet the needs of dynamic expansion. Some existing technologies attempt to introduce automated matching algorithms to establish mapping relationships between unknown data and standard fields. However, most of these methods only perform local similarity calculations based on a single data dimension, such as comparing only the name strings of fields or simple fluctuations in numerical values. This approach ignores the spatial hierarchical nesting relationships between nodes within the data packet and the inherent physical business logic behind the fields. When processing complex multi-source sensor data, it may incorrectly associate fields with similar numerical characteristics but completely different actual physical meanings, resulting in mapping conversion accuracy that fails to meet production requirements.

[0004] Existing data governance systems, when faced with massive concurrent inflows of sensor data, employ a one-size-fits-all processing approach. This involves performing complete syntax tree parsing and feature comparison on all received data packets. This approach fails to distinguish between regular steady-state packets with unchanged data structures and abnormal packets with altered structures. Consequently, the system consumes excessive and unnecessary computational and memory resources when processing a large number of duplicate packets with known structures, resulting in low overall concurrent processing efficiency and making it difficult to support real-time data aggregation and standardized governance tasks at the city level. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a standardized governance method for multi-source heterogeneous data in city-level smart communities. This method solves the problems of low accuracy, high computational cost, and lack of environmental adaptability in data standardization conversion caused by reliance on manual configuration or single-dimensional matching when facing dynamic changes in the underlying device data structure.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a standardized governance method for multi-source heterogeneous data in city-level smart communities, comprising the following steps: Heterogeneous data packets are acquired, a structural topological fingerprint of the heterogeneous data packets is generated, and differential processing is performed on the heterogeneous data packets based on the comparison results of the structural topological fingerprint. The differential processing includes: when the comparison results are inconsistent, extracting the temporal statistical features of unknown fields in the heterogeneous data packets to construct physical feature vectors, and calculating the topological structural correlation between the unknown fields to construct a topological distance matrix; based on the physical feature vectors and preset standard field reference vectors, combined with the topological distance matrix, constructing and solving a global mapping loss function to establish the mapping relationship between the unknown fields and standard fields; based on the mapping relationship and the network communication protocol type to which the heterogeneous data packets belong, generating a streaming mask for locating target fields, and extracting and storing standardized data based on the streaming mask.

[0007] In a preferred embodiment of the present invention, structural topological fingerprints are used as the determination criterion for data flow status to identify changes in message structure; for data with structural changes, physical features in the time series dimension and topological features in the message spatial hierarchy dimension are jointly extracted; feature space distance and topological cross-penalty terms are introduced into the global mapping loss function, and the mapping relationship is constrained by combining data quantization features and spatial distribution relationships to exclude mapping terms with similar single-dimensional features but contradictory topological logic; finally, a memory extraction mask is dynamically generated according to the protocol type to achieve standardized extraction and storage of heterogeneous data.

[0008] Furthermore, the step of generating the structural topology fingerprint of the heterogeneous data packet specifically includes: parsing the byte stream of the heterogeneous data packet into an abstract syntax tree; performing lexicalization on the abstract syntax tree, removing key name data and numerical data, retaining data type tags and hierarchical nesting relationships, and generating a pure structured abstract syntax tree; serializing the pure structured abstract syntax tree into a one-dimensional pure structured sequence; and performing a hash operation on the one-dimensional pure structured sequence to generate the structural topology fingerprint.

[0009] In a preferred embodiment of the present invention, the word-de-lexical processing removes business semantics and numerical variables, while retaining data type markers and hierarchical nesting relationships to characterize the basic structural features of the message; a hash operation is performed on the serialized one-dimensional pure structured sequence to generate a fixed-length and unique structural topology fingerprint, which is used to perform fast comparison and matching of message structures in memory.

[0010] Furthermore, the step of extracting the time-series statistical features specifically includes: within a sliding time window of a preset length, calculating the data reporting frequency, numerical time-series variance, classification cardinality, and data sparsity for the unknown field; normalizing the calculated data reporting frequency, numerical time-series variance, classification cardinality, and data sparsity, and combining them into the physical feature vector.

[0011] In a preferred embodiment of the present invention, the data value characteristics at a single moment are relatively simple. The reporting frequency, variance, cardinality and sparsity are extracted within the sliding time window to quantify the time-series statistical change characteristics of the data field. Normalization is used to eliminate the dimensional differences between different statistical items and to unify the calculation weights of each indicator in the physical feature vector.

[0012] Furthermore, the step of calculating the topological correlation between the unknown fields specifically includes: based on the abstract syntax tree generated by parsing, calling a graph theory search algorithm to calculate the number of edges traversed by the shortest path between the leaf nodes corresponding to any two unknown fields; and using the number of edges as matrix elements to construct the topological distance matrix.

[0013] In a preferred embodiment of the present invention, the number of edges crossed by the shortest path between leaf nodes is calculated based on the abstract syntax tree, which can objectively characterize the degree of hierarchical association of each field in the data encapsulation structure in heterogeneous messages, and supplement the feature data of the spatial structure dimension by constructing a topological distance matrix.

[0014] Furthermore, the step of constructing and solving the global mapping loss function includes: calling the matrix operation unit, based on the Mahalanobis distance calculation rule, and introducing the inverse matrix of the feature space covariance matrix, to calculate the basic physical feature Mahalanobis distance between the physical feature vector of the unknown field and the reference vector of each standard field.

[0015] In a preferred embodiment of the present invention, the inverse matrix of the feature space covariance matrix is ​​introduced to calculate the Mahalanobis distance, which is used to eliminate collinearity interference within the multidimensional time series feature vectors and calculate the difference in physical feature distribution between the unknown field and the standard field.

[0016] Furthermore, the step of constructing and solving the global mapping loss function includes: combining the topological distance matrix and the preset standard field prior physical correlation coefficient matrix to calculate and generate a topological cross-penalty term for characterizing spatial association logic; and combining and adding the basic physical feature Mahalanobis distance with the topological cross-penalty term to construct the global mapping loss function.

[0017] In a preferred embodiment of the present invention, the topological distance matrix is ​​combined with the prior physical correlation coefficient matrix to generate a topological cross penalty term, which serves as a constraint on the mapping relationship. This penalty term is introduced when constructing the global mapping loss function, so that the mapping calculation is simultaneously constrained by the physical feature distance and the topological spatial correlation, thereby eliminating erroneous mappings between fields with no physical correlation.

[0018] Furthermore, the step of solving the global mapping loss function includes: calling a heuristic matching algorithm to iteratively calculate the global mapping loss function until the overall loss value reaches a preset convergence state, searching and outputting a set of mapping relationships that minimize the overall loss value, as the mapping relationship.

[0019] In a preferred embodiment of the present invention, a heuristic algorithm is used to perform global iterative optimization of the global mapping loss function, and the allocation scheme corresponding to the minimum value of the output function loss is used to establish the optimal set of overall mapping relationships from unknown fields to standard fields, thereby avoiding the decline in overall mapping accuracy caused by local matching errors.

[0020] Furthermore, the step of generating the streaming mask includes: when the heterogeneous data packet is determined to be a binary protocol packet with a fixed-length encoding mechanism, calculating the fixed absolute byte offset of the target field in the memory block according to the mapping relationship and the data type byte length, and generating a static absolute displacement mask as the streaming mask.

[0021] In a preferred embodiment of the present invention, taking into account the fixed structure of the fixed-length binary protocol, the absolute byte offset of the target field is obtained by accumulating the preset byte length of the data type of the preceding node, and a static mask is generated accordingly to directly locate the address space of the target field during the memory reading stage.

[0022] Furthermore, the step of generating the streaming mask includes: when the heterogeneous data packet is determined to be a text protocol packet with a variable length encoding mechanism, extracting the structural symbol step vector required for the target field to traverse from the root node to the target address, and calling the single instruction stream multiple data stream extended instruction set to perform parallel scanning on the packet memory block to generate a single instruction stream multiple data stream topology projection mask as the streaming mask.

[0023] In a preferred embodiment of the present invention, in view of the variable length text protocol field with an unfixed length, a single instruction stream multiple data stream instruction set is used to perform concurrent scanning matching based on the structure symbol step vector in order to dynamically locate and extract the start and end memory address boundaries of the target field.

[0024] Furthermore, when the comparison result of the structural topology fingerprint is consistent, according to the preset associated execution path in the historical fingerprint routing table, the heterogeneous data packet is controlled to directly flow to generate a streaming mask based on the mapping relationship and the network communication protocol type to which the heterogeneous data packet belongs, and then extract and store it.

[0025] In a preferred embodiment of the present invention, when the structural topology fingerprint matches, it is determined that the current message structure has not changed. The existing mapping relationship is directly reused to generate a mask and perform data extraction, skipping the time-series feature extraction and loss function calculation, thereby reducing the data processing overhead of the computing center and improving data transmission efficiency.

[0026] This invention provides a standardized governance method for multi-source heterogeneous data in city-level smart communities. It has the following beneficial effects: 1. This invention extracts the temporal statistical features of unknown fields in a message and constructs a topological distance matrix based on an abstract syntax tree to build a global mapping loss function that includes Mahalanobis distance of basic physical features and a topological cross-penalty term. This design introduces spatial structure association as a constraint while quantifying the differences in the distribution of physical features of data, and eliminates local mapping combinations with similar single-dimensional features but contradictory spatial topological logic, thereby improving the accuracy of mapping and converting heterogeneous data to standard fields.

[0027] 2. This invention generates a structural topology fingerprint by performing word-scraping processing on data packets and hashing operations. The comparison result of this fingerprint is used as the basis for scheduling and determining the data flow processing path. When the data structure of the underlying device undergoes unknown changes, the system can automatically detect the changes through the fingerprint inconsistency comparison results and autonomously trigger the spatiotemporal joint feature extraction and mapping relationship reconstruction process, replacing the traditional manual rule pre-configuration process and improving the system's adaptive ability to cope with the dynamic updates of complex heterogeneous sensor network data.

[0028] 3. This invention generates static absolute displacement masks or single-instruction stream multi-data stream topology projection masks according to the network communication protocol type, and performs data flow by combining the associated execution path when the fingerprint is consistent. For known messages with unchanged structure, the system directly reuses the mapping relationship and the corresponding mask to perform data boundary positioning and truncation at the memory level, eliminating the need for repeated message structure parsing and feature calculation steps, reducing the computing power overhead of computing nodes, and ensuring the system processing efficiency when massive concurrent data access is available. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the physical architecture of the city-level multi-source heterogeneous data standardization governance system of the present invention. Figure 2 This is a schematic diagram of the workflow of the present invention; Figure 3This is a timing diagram of the structural topology fingerprint extraction and verification process of the present invention; Figure 4 This is a schematic diagram of the spatiotemporal joint feature acquisition and matrix construction process of the present invention; Figure 5 This is a schematic diagram of the multidimensional feature mapping process based on topological constraints of the present invention. Figure 6 This is a schematic diagram of the dynamic protocol awareness and streaming mask extraction process of the present invention. Figure 7 This is a bar chart showing the heterogeneous field matching accuracy of the present invention. Figure 8 This is a bar chart showing the peak CPU utilization rate of the system according to the present invention. Detailed Implementation

[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] See attached document Figure 1 To support the standardized governance method for multi-source heterogeneous data in a city-level smart community provided by this invention, the system includes a corresponding hardware support environment to provide the physical foundation for data access, logical processing and data storage. The system mainly includes a data access module, a logical processing module and a data storage module.

[0032] The data access module is deployed in the data acquisition area at the front end of the smart community. It is equipped with a communication control interface and a microprocessor. The data access module maintains a network connection with various sensing devices and the underlying low-voltage control system in the community. It receives multi-source heterogeneous underlying network communication messages and transmits the aggregated raw message data to the logic processing module. The logic processing module is deployed in the computing center and includes multiple computing nodes equipped with multi-core processors and memory arrays. The logic processing module is configured with a single instruction stream multiple data stream extended instruction set runtime environment, which constitutes the computing power carrier for performing logical operations. The data storage module includes multiple distributed storage servers, which are connected to the logic processing module through a high-speed network interface. They are used to receive and persistently store the standardized structured data. The above hardware devices work together to form the underlying execution environment of the method of this invention.

[0033] See attached document Figure 2 This invention provides a standardized governance method for multi-source heterogeneous data in city-level smart communities, comprising the following steps: The logic processing module receives heterogeneous data packets from the data access module in real time. Within the memory region, the logic processing module invokes a lexical analysis operator to parse the serialized packet byte stream into an abstract syntax tree. During the parsing operation, the logic processing module separates and removes the key name data and leaf node numerical data from the packet nodes, retaining the data type markers and the hierarchical nesting relationships of the tree structure. The logic processing module then uses a hash operation unit to perform encrypted calculations on the retained tree structure, generating a structural topology fingerprint of the heterogeneous data packet.

[0034] The logic processing module performs a hash comparison between the generated structural topology fingerprint and the historical fingerprint routing table stored in memory. If the comparison matches, the logic processing module controls the data flow to directly proceed to the process of determining the message protocol type and performing mask extraction and disk storage. If the comparison results do not match, the logic processing module triggers a bypass routing instruction, cutting off the direct processing path of the current data flow, scheduling the data flow to the spatiotemporal feature acquisition buffer in memory, and establishing a sliding time window in the buffer to perform time-series statistics and spatial topology calculations for unknown fields. The logic processing module establishes a sliding time window of a preset length within the spatiotemporal feature acquisition buffer. For heterogeneous message data streams within the buffer, the logic processing module extracts each unknown field that has not established a mapping relationship within the sliding time window and performs time series statistics. The statistical items specifically include data reporting frequency, numerical time series variance, classification cardinality, and data sparsity. The logic processing module combines the statistical values ​​of each unknown field to construct the corresponding physical feature vector.

[0035] Within the same time window period, the logic processing module, based on the abstract syntax tree structure obtained by parsing and converting the message byte stream, calls the multi-core processor to run the graph theory search algorithm to calculate the number of edges crossed by the shortest path between any two unknown leaf nodes in the abstract syntax tree. The logic processing module uses the edge count calculation result as matrix elements to construct a topological distance matrix with zero elements on the main diagonal. The logic processing module obtains the standard field reference feature matrix from the data storage module through the network interface. The logic processing module calls the internal matrix operation unit to calculate the feature space distance between the physical feature vector of the unknown field and the reference vector of each standard field according to the Mahalanobis distance calculation rule. The logic processing module combines the topological distance matrix and the preset standard field prior physical correlation coefficient matrix to establish a cross-mapping relationship equation containing the topological space penalty term, and constructs a global mapping loss function accordingly. The logic processing module calls the heuristic matching algorithm to iteratively calculate the global mapping loss function and search for the mapping relationship set that minimizes the overall loss value. This mapping relationship set establishes the correspondence between the unknown field at the input end and the built-in fields of the data storage module. The logic processing module receives the mapping relationship set that establishes the correspondence between the input end and the data storage module, which is output by the iterative optimization search of the global mapping loss function. It then determines the network communication protocol type to which the heterogeneous data packet belongs. For binary protocol packets using a fixed-length encoding mechanism, the logic processing module calculates the fixed absolute byte offset of the packet in the memory block based on the mapping relationship and the data type byte length, and generates a static absolute displacement mask accordingly.

[0036] For text protocol messages using variable-length encoding, the logic processing module extracts the abstract syntax tree structure generated by the pre-parsed message transformation, calculates the number of structural symbol steps the message needs to traverse from the starting memory address to the target field address, and generates a single-instruction-stream multi-data-stream topology projection mask based on this. The logic processing module loads the generated mask rules into the underlying computation operator, which locates the target field boundary in the memory byte sequence according to the mask rules, performs truncation and memory copy operations, and then converts and assembles the extracted data into a table structure format that meets the requirements of the data storage module. Finally, it transmits the data over the network to the storage node for disk storage.

[0037] See attached document Figure 3 , Figure 3 This is a timing diagram of the structural topology fingerprint extraction and verification process according to an embodiment of the present invention. In this embodiment, the logic processing module receives the original byte stream sequence from the data access module through the network interface and allocates a dedicated buffer in the memory area. The logic processing module calls the lexical analysis operator to scan the original byte stream sequence byte by byte to identify the boundaries between control characters and data entities. The system thereby converts the one-dimensional byte stream into an initial abstract syntax tree with a tree structure. After the tree structure is constructed, the logic processing module performs a delexification operation on the initial abstract syntax tree. The logic processing module traverses all nodes and strips the key name data containing business semantics and the specific numerical data carried by the leaf nodes. As a preferred method, after the stripping operation, only the data type mark representing the data format and the tree depth information reflecting the hierarchical nesting relationship and the parent-child node connection edge are retained. The data type mark is set to one of integer, floating point, boolean, string or array type. The system thereby obtains a pure structured abstract syntax tree.

[0038] The logic processing module formulates serialization encoding rules for the pure structured abstract syntax tree. The system uses a depth-first search traversal algorithm to sequentially concatenate and map the retained data type tags and hierarchical nesting relationships into a one-dimensional pure structured sequence. In the serialization process of this embodiment, the depth value of the tree structure level where the node is located is used as a prefix and concatenated with the data type tag to form the structural identifier of the independent node. The structural identifiers of all nodes are connected in traversal order to form a pure structured sequence. The logic processing module calls the hash operation unit to perform one-way encrypted digest calculation on the pure structured sequence using a secure hash algorithm. The operation unit outputs a fixed-length string as the structural topology fingerprint of the current heterogeneous data packet. The specific execution logic of the one-way digest calculation is based on existing hash mapping calculation methods. The mathematical expression relationship of the relevant calculation is as follows.

[0039] ; In the formula: To calculate the structural topology fingerprint string corresponding to the generated heterogeneous data packets; This is the one-way encrypted mapping function corresponding to the secure hash algorithm; It is a one-dimensional pure structured sequence generated after word removal processing.

[0040] After generating the structural topology fingerprint, the logic processing module reads the historical fingerprint routing table that records the structure information of past data packets within its configured high-speed cache memory area. The logic processing module performs memory-state data matching and comparison between the currently generated structural topology fingerprint and the fingerprint records in the historical fingerprint routing table. When the comparison results are consistent, the logic processing module determines that the structure of the currently accessed data stream has not changed. The system maintains the original transmission link of the current data stream and directly flows to the subsequent steady-state extraction stage according to the preset associated execution path instructions in the routing table. When the comparison results are inconsistent, the logic processing module determines that the current packet has internal structural changes. As a preferred method, the logic processing module generates and sends a bypass route trigger instruction to the scheduling controller. The system's underlying scheduling module cuts off the original direct transmission link of the current data stream according to the instruction. The logic processing module smoothly transfers the packet data stream carrying the unmatched structural topology fingerprint to the pre-allocated spatiotemporal feature acquisition buffer for memory isolation and temporary storage, and queues it for subsequent feature calculation stages.

[0041] See attached document Figure 4 , Figure 4This is a schematic diagram of the spatiotemporal joint feature acquisition and matrix construction process according to an embodiment of the present invention. In this embodiment, the logic processing module performs the joint feature extraction operation of physical features and spatial structure. Single-dimensional data features face the risk of dimension confusion in heterogeneous environments. By combining the physical features of the time series dimension with the spatial topology reflected by the abstract syntax tree, a joint feature benchmark is constructed to provide multi-dimensional judgment basis. The logic processing module establishes a sliding time window in the spatiotemporal feature acquisition buffer according to the system clock. The system allocates a dedicated time sequence record queue as the first unknown structure message received as a trigger signal. The logic processing module continuously pushes the arriving messages into the queue. As a preferred method, the time length threshold and capacity limit are calibrated based on the physical memory configuration of the edge gateway and the historical average network throughput. The value range of the time length threshold is set to 500 to 2000 milliseconds, and the value range of the capacity limit is set to 1000 to 5000 message samples. When any threshold is triggered, the system sends a signal to close the sliding time window.

[0042] After the time window is closed, the logic processing module performs time-series dimension statistical quantification on each unknown field in the record queue. The logic processing module calls the accumulation operator to divide the total number of valid values ​​received in the window by the time span to calculate the data reporting frequency. The system calculates the data sparsity by the ratio of the number of samples with null or zero values ​​in the statistical fields to the total number of samples. The logic processing module calls the hash set deduplication algorithm to compare all non-null valid values ​​to obtain the number of independent categories as the classification cardinality. The system employs an online variance statistics algorithm to calculate the numerical time-series variance of numerical samples, representing the degree of deviation from the mean. For non-numerical samples such as Boolean or string types, the system uses the state switching frequency of these samples as a substitute, or fills the feature values ​​of that dimension with zeros by default to ensure the completeness of the multi-dimensional vector structure. In this embodiment, the logic processing module uses a maximum-minimum normalization algorithm to perform a dimensionless conversion operation on the above four statistical values. The system then linearly concatenates the converted values ​​in a fixed order to assemble a physical feature vector representing the dynamic behavior pattern of the field. The specific mathematical equation of the physical feature vector is objectively expressed through feature combination. ; In the formula: A column vector of physical features constructed for unknown fields; The data reporting frequency value calculated for this unknown field; The time series variance of the value calculated for this unknown field; The numerical value corresponding to the classification cardinality calculated for this unknown field; The percentage of data sparsity calculated for this unknown field; This is the identifier for the transpose operation of a vector. This is the sort index value corresponding to the currently unknown field in the abstract syntax tree.

[0043] Within the same period after completing the temporal feature extraction, the logic processing module performs spatial dimension correlation analysis based on the abstract syntax tree generated in the previous steps. The system sets the weight of the connection edges between all parent and child nodes in the syntax tree to an equal unit value. The logic processing module calls the graph theory breadth-first search algorithm and allocates independent search queue resources in memory. The system pushes any unknown leaf node into the queue as the starting point and performs backtracking and downward traversal on the parent node and adjacent branches. The system continuously accumulates the number of connection edges crossed according to the hierarchical progression until the target unknown leaf node is reached. The logic processing module outputs the total number of edges crossed by the shortest path recorded in the traversal as the absolute topological distance. This breadth-first search mechanism avoids the risk of path dead loops in complex tree structures and ensures the complete execution of the shortest path calculation.

[0044] The logic processing module traverses all unknown leaf node pairs contained in the abstract syntax tree and extracts the absolute topological distance between nodes in turn. The system maps the unknown leaf nodes to the row and column indices of a two-dimensional memory array in a fixed increasing sequence. The logic processing module writes the extracted absolute topological distance values ​​into the corresponding cross coordinates of the array to construct a topological distance matrix. As a preferred method, this topological distance matrix presents a real symmetric matrix with all diagonal elements being zero in the underlying data structure. The mathematical mapping relationship of the matrix construction is as follows.

[0045] ; In the formula: These are the absolute topological distance values ​​recorded in the topological distance matrix; This is the pathfinding distance calculation function corresponding to the breadth-first search algorithm; The sorting index in the abstract syntax tree is Unknown leaf nodes; The sorting index in the abstract syntax tree is Unknown leaf nodes; This is the sort index value corresponding to the currently unknown field in the abstract syntax tree; This is the sort index value corresponding to another parallel unknown field in the abstract syntax tree.

[0046] See attached document Figure 5 , Figure 5This is a schematic diagram of a multidimensional feature mapping process based on topological constraints according to an embodiment of the present invention. In this embodiment, the logic processing module sends a read instruction to the data storage module through the network interface to load the preset standard field reference feature vector and feature space covariance matrix. The logic processing module extracts the pre-constructed physical feature vector in the internal matrix operation unit and establishes a preliminary mapping candidate set based on the data type. Data type matching is used as a preprocessing condition to automatically filter invalid combinations with type conflicts in memory. For each unknown field and candidate standard field combination, the logic processing module calls the underlying instruction to calculate the Mahalanobis distance between the two vectors. As a preferred method, the logic processing module introduces the inverse matrix of the feature space covariance matrix to perform multiplication operations in the calculation. This operation integrates the distribution differences of different feature dimensions and eliminates the numerical interference of multicollinearity. The system thus outputs a basic distance quantification index reflecting the physical similarity between unknown nodes and known nodes in the multidimensional feature space.

[0047] ; In the formula: The calculated Mahalanobis distance value is the basic physical characteristic between the unknown field and the standard field. This is the column vector of physical features corresponding to the unknown fields; This refers to the reference feature column vector corresponding to the standard fields; This is an identifier for the transpose operation of a matrix or vector; It is the inverse matrix corresponding to the covariance matrix of the feature space; This is the ranking index value corresponding to the currently unknown field or unknown leaf node extracted from the abstract syntax tree; This refers to the sorting index value corresponding to the candidate criterion field in the data storage module.

[0048] After obtaining the basic distance index, the logic processing module reads the standard field prior physical correlation coefficient matrix maintained in the data storage module. This matrix is ​​generated based on the Pearson correlation coefficients of various standard fields in historical steady-state data and records the linkage change law of physical quantities. The logic processing module extracts the spatial edge number value from the topological distance matrix generated in the previous step. The system fuses the spatial topological variables and the prior physical correlation coefficient variables to construct a topological cross penalty term for a specific node mapping combination. In this embodiment, the system introduces a joint continuous algebraic equation containing the complement of topological distance and correlation coefficient. The mathematical properties of this equation objectively make it possible that when the topological distance between two unknown fields in the syntax tree is less than a certain range, the preset threshold of the specific range as an observation boundary can be set to an integer between 2 and 4. When the current mapping scheme attempts to map the two to independent standard fields with correlation coefficients close to zero, a numerical amplification effect is automatically generated. The calculation operator directly calculates and assigns the penalty value corresponding to this mapping combination according to the inverse proportional function form. The system relies on the penalty function to eliminate pseudo-mapping combinations with similar basic physical characteristics but spatial correlation logic that deviates from prior common sense. ; In the formula: The topological crossover penalty value calculated when mapping unknown fields to specific standard fields; The penalty adjustment weight coefficient is pre-configured for the system, and its value range is set between 0.5 and 2.0; This represents the absolute topological distance between two unknown fields recorded in the topological distance matrix. The correlation coefficient value between two standard fields recorded in the prior physical correlation coefficient matrix, with the value range being negative one to positive one; This is the absolute value corresponding to the correlation coefficient. This is the sort index value corresponding to the currently unknown field in the abstract syntax tree; This refers to the sorting index value corresponding to the candidate criterion field in the data storage module; This is the sort index value corresponding to another parallel unknown field in the abstract syntax tree; This is the sorting index value corresponding to another parallel standard field in the data storage module.

[0049] After establishing the local quantization method, the logic processing module calls the accumulator to combine and add the Mahalanobis distance of all candidate mapping node pairs with the accompanying topological cross penalty term. The system uses this to establish a global mapping loss function that covers the global node mapping state of the current message. The logic processing module calls the bipartite graph matching heuristic algorithm in the memory area to perform iterative optimization on the loss function. As a preferred approach, the logic processing module sets an upper limit on the number of steps in the loop based on hardware computing power. In this embodiment, the upper limit of the number of steps is set to a range of 500 to 2000 times. This is combined with a convergence threshold based on system accuracy. In this embodiment, the convergence threshold is set to a range of 0.001 to 0.01. As a dual control condition, when the number of iterations reaches the upper limit or the absolute value of the descent gradient of the loss function for three consecutive iterations is less than the convergence threshold, the logic processing module determines that the system has reached the minimum convergence state. The system then interrupts the iteration and outputs the global allocation array with the minimum current loss value. If there are isolated unknown fields that cannot be matched, the system marks them and transfers them to the abnormal observation queue. This allocation array ultimately establishes a unique correspondence between the input end and the storage end.

[0050] ; In the formula: This represents the total value of the global mapping loss function corresponding to the current mapping combination; This is a set of global mapping relationships generated within the current iteration cycle; This is the sort index value corresponding to the currently unknown field in the abstract syntax tree; This refers to the sorting index value corresponding to the candidate criterion field in the data storage module; A matching pair in a mapping relationship set that maps a single unknown field to a specific standard field; This is the sort index value corresponding to another parallel unknown field in the abstract syntax tree; This is the sorting index value corresponding to another parallel standard field in the data storage module; For another set of unknown fields to standard fields that are matched in parallel within the mapping relationship set; The Mahalanobis distance values ​​are calculated based on the basic formula to match the pairs; The topological crossover penalty value calculated when mapping unknown fields to specific standard fields.

[0051] See attached document Figure 6 , Figure 6This is a schematic diagram of the dynamic protocol awareness and streaming mask extraction process according to an embodiment of the present invention. In this embodiment, after obtaining a determined mapping array, the logic processing module performs low-level parsing and scheduling on the input data stream. The logic processing module extracts the communication identifier field from the header of the heterogeneous data packet and reads the root node attribute of the abstract syntax tree. The system performs a lookup and matching operation on the extracted identifier based on the protocol feature dictionary that is pre-configured and stored in memory. The system outputs the network communication protocol category to which the current data stream belongs based on the matching result. As a preferred method, the system receives the identification result and activates the hardware-level routing distribution controller. When it is determined to be a binary protocol using a fixed byte length encoding mechanism, the routing distribution controller pushes the current packet and mapping array to the processing pipeline where the fixed-length mask generation operator is located. When it is determined to be a text protocol using dynamic length encoding, the controller switches the data stream into the execution pipeline of the variable-length mask generation operator. The system allocates the heterogeneous data stream to the corresponding physical memory encoding parsing channel.

[0052] After receiving scheduling data, the fixed-length mask generation operator automatically removes isolated nodes that have not completed matching and extracts the corresponding preceding node sequence only for target fields with successfully established unique mapping relationships. The logic processing module reads the standard byte length rules for each data type preloaded in the system configuration table, accumulates the preset data type byte lengths of all preceding nodes, and calculates the absolute byte offset of the target field in a contiguous memory block. The system assembles the absolute byte offset with the type length of the target field itself to generate a static absolute displacement mask. The underlying calculation operator loads this static absolute displacement mask and obtains the starting address pointer of the current packet in memory. The operator directly performs an addition operation between the starting address pointer and the absolute byte offset to obtain the shifted memory read position. The system sends parameters to the direct memory access controller according to the target field type length. The direct memory access controller copies a contiguous byte block of the corresponding length from the shifted pointer position to the output buffer. The system performs memory-level data extraction operations based on the underlying hardware control path. ; In the formula: This is the calculated absolute byte offset value; The traversal index variable for the preceding nodes that precede the target field; The index value for the order of the target field within the fixed-length protocol message; The first one before the target field The data type byte length pre-configured for each front-end node.

[0053] When the variable-length mask generation operator starts execution, it automatically filters out isolated nodes without established mapping relationships and retrieves the node association information inside the pre-generated pure structured abstract syntax tree. The logic processing module extracts the number of structure symbols that need to be traversed from the root node to the successfully matched target field along the tree structure and generates a structure symbol step vector representing the depth of the hierarchy. The system calls the single instruction stream multiple data stream extended instruction set integrated inside the multi-core processor to load the structure symbol dictionary as a matching template into the wide byte vector register. As a preferred method, the logic processing module controls the extended instruction set to perform parallel byte scan comparison on the variable-length message memory block. The hardware logic unit maps the location of the structure symbols in the message to binary Boolean state through parallel comparison instructions and outputs it to an independent memory space. The system splices and combines the parallel output Boolean state sequence into a position index bitmap that precisely corresponds to the physical length of the original message.

[0054] After generating the location index bitmap, the logic processing module calls a hardware-level counter instruction to scan and accumulate the bitmap sequence. When the accumulated value matches the value recorded in the structure symbol step vector, the system extracts the starting and ending memory addresses of the target variable-length data as boundary references. If no match is found after scanning to the end of the bitmap, the system triggers an abnormal blocking mechanism to discard the current packet and record it in the system log. Conversely, if the boundary reference is successfully established, the underlying operator sets the start and end reference pointers in memory based on the determined address boundaries and performs in-place truncation without copying to generate a discrete variable-length byte stream. The system performs field filling and format conversion processing on all extracted byte streams according to the standard relational table structure specifications configured by the data storage module. The logic processing module encapsulates and combines the assembled standard structure data stream into a continuous data block array. The system drives the high-speed network interface protocol stack to stream and pass the standardized data blocks through and write them to the physical disk media of the underlying distributed storage service server for disk storage.

[0055] To aid in understanding the technical solution of this invention, the following provides an application example based on heterogeneous data governance of IoT devices in a city-level smart community.

[0056] In this application example, the network access device in the front-end data acquisition area is deployed at the computing node of the smart community data center. The network access device in the front-end data acquisition area is connected to the underlying sensing unit and the low-voltage control system. The above devices output access control status communication protocol messages using fixed-length binary encoding format and smart water and electricity metering IoT protocol messages using variable-length text encoding format in parallel.

[0057] When a smart metering terminal performs a firmware update, causing changes to the internal structure of the output variable-length text message and adding a device power status data node before the original metering reading data entity, the logic processing module receives the changed message byte stream in real time and calls the lexical analysis operator to convert the serialized message into an initial abstract syntax tree. The logic processing module performs delexification processing on the initial abstract syntax tree, stripping the numerical data and business key names carried by each node, retaining the data type markers and the hierarchical nesting relationship containing tree depth information and parent-child node connection edges to generate a pure structured abstract syntax tree. The logic processing module uses a depth-first search traversal algorithm to concatenate and map the retained features into a pure structured sequence, and calculates and outputs the structural topology fingerprint of the current message through the hash operation unit.

[0058] The logic processing module performs a memory-state comparison between the currently generated topological fingerprint and the historical fingerprint routing table records in the cache, and determines that the comparison results are inconsistent. The underlying scheduling module cuts off the original direct link of the current data stream according to the bypass routing trigger instruction issued by the logic processing module, and schedules the message carrying the new power node to the spatiotemporal feature acquisition buffer for memory isolation. The logic processing module establishes a sliding time window with a length of 1000 milliseconds in the buffer based on the system clock, extracts the unknown fields of the new power that have not established a mapping relationship, and calls the accumulation operator and online variance statistics algorithm to quantize the execution time sequence dimension. Within the same period, the system calculates the number of edges crossed by the shortest path between the unknown node and other adjacent unknown leaf nodes in the pure structured abstract syntax tree based on the breadth-first search algorithm, and constructs a topological distance matrix with all diagonal elements being zero.

[0059] The logic processing module calculates the Mahalanobis distance between the physical feature vector constructed from the newly added electricity field and the standard feature reference vector in the data storage module by combining the inverse matrix of the feature space covariance matrix through the internal matrix operation unit. The logic processing module introduces the topological distance matrix and the prior physical correlation coefficient matrix to generate the topological cross penalty term. The system combines the topological cross penalty term with the Mahalanobis distance to establish a global mapping loss function. The system calls the bipartite graph matching heuristic algorithm to perform iterative optimization on the loss function. The system reduces the total value of the loss function to a convergent state and establishes the mapping relationship between the newly added field at the input end and the standard electricity field at the storage end.

[0060] After determining that the current message uses a variable-length text protocol, the logic processing module extracts the above-mentioned abstract syntax tree structure and calculates the structural symbol step vector from the root node to the power target field. The logic processing module loads the underlying structural symbol dictionary into the wide-byte vector register of the single-instruction stream multiple-data stream extended instruction set. The system performs parallel byte scanning on the message memory block to generate a position index bitmap. The system calls hardware-level instructions to scan the bitmap, accumulate values, and determine the start and end absolute memory addresses of the power target field. The underlying operator sets reference pointers to perform in-place copy-free truncation operations. The system extracts the truncated discrete byte stream and assembles, fills, and passes it through to the distributed storage server according to the relational table structure format for physical disk storage.

[0061] Table 1: Performance Comparison Test Data of Heterogeneous Message Parsing and Mapping Systems According to Table 1 and Figure 7 and Figure 8 As can be seen, the standardized governance method for multi-source heterogeneous data in city-level smart communities provided by this invention improves processing efficiency when dealing with heterogeneous data streams. The traditional structure delimitation matching test group, which runs based on fixed regular expressions and static memory offsets to cope with changes in the front-end device structure, has a heterogeneous field matching accuracy of 32.18% and generates dropped messages. The single-dimensional semantic dictionary comparison test group, which compares heterogeneous fields based on character content similarity, has a heterogeneous field matching accuracy of 74.31%. The single-dimensional semantic dictionary comparison test group has mismatch phenomena due to the lack of a verification mechanism for the spatial connection relationship of nodes and the time series running status. The full tree depth traversal operation caused by the single-dimensional semantic comparison process causes the system CPU peak utilization rate to reach 65.82% and the average processing delay of mutated messages to be 147.25 milliseconds.

[0062] The heterogeneous field matching accuracy of the method of this invention is 98.63%. The logic processing module executes the spatiotemporal joint feature acquisition and matrix construction process within the sliding time window. The system assembles the data reporting frequency, numerical temporal variance and classification cardinality into physical feature vectors. The system constructs matrix variables by extracting the absolute topological distance of the abstract syntax tree. The calculation operator introduces the feature space covariance matrix according to the Mahalanobis distance rule and combines it with the prior physical correlation coefficient matrix of the standard field to generate a cross-penalty term. The system imposes constraints on the matching candidate relationship in the non-semantic dimension through the physical feature vector and matrix variables and eliminates combinations with similar feature distances but conflicting topological logic.

[0063] The method of this invention has an average processing latency of 16.84 milliseconds for mutated messages and a data normalization throughput of 114.62 MB / s. When the structural topology fingerprint is inconsistent, the bypass routing instruction isolates and temporarily stores the unknown structure message. After determining the mapping relationship, the system performs parallel state scanning of wide-byte memory blocks according to the single instruction stream multiple data stream extended instruction set. After calculating the address boundary, the underlying operator sets the memory pointer reference to perform in-place copy-free truncation operation. This memory control strategy reduces the processor clock cycle occupancy during the processing cycle, and the system's peak CPU utilization rate drops to 22.47%.

[0064] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A standardized governance method for multi-source heterogeneous data in city-level smart communities, characterized in that, Includes the following steps: Obtain heterogeneous data packets, generate structural topological fingerprints of the heterogeneous data packets, and perform differential processing on the heterogeneous data packets based on the comparison results of the structural topological fingerprints; The differentiation process includes: when the comparison results are inconsistent, extracting the temporal statistical features of unknown fields in the heterogeneous data packets to construct a physical feature vector, and calculating the topological correlation between the unknown fields to construct a topological distance matrix. Based on the physical feature vector and the preset standard field reference vector, combined with the topological distance matrix, a global mapping loss function is constructed and solved to establish the mapping relationship between the unknown field and the standard field; Based on the mapping relationship and the network communication protocol type of the heterogeneous data packets, a streaming mask for locating target fields is generated, and standardized data is extracted and stored based on the streaming mask.

2. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 1, characterized in that, The step of generating the structural topological fingerprint of the heterogeneous data packet includes: The byte stream of the heterogeneous data message is parsed into an abstract syntax tree; The abstract syntax tree is subjected to delexification, key name data and numerical data are removed, data type tags and hierarchical nesting relationships are retained, and a pure structured abstract syntax tree is generated. The pure structured abstract syntax tree is serialized into a one-dimensional pure structured sequence; A hash operation is performed on the one-dimensional pure structured sequence to generate the structural topological fingerprint.

3. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 1, characterized in that, The step of extracting the time-series statistical features includes: Within a preset sliding time window, for the unknown field, the data reporting frequency, numerical time series variance, classification cardinality, and data sparsity are calculated respectively. The calculated data reporting frequency, numerical time-series variance, classification cardinality, and data sparsity are normalized and combined into the physical feature vector.

4. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 1, characterized in that, The steps for calculating the topological correlation between the unknown fields include: Based on the abstract syntax tree generated by parsing, a graph theory search algorithm is called to calculate the number of edges crossed by the shortest path between any two leaf nodes corresponding to unknown fields; The topological distance matrix is ​​constructed by using the number of edges as matrix elements.

5. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 1, characterized in that, The step of constructing and solving the global mapping loss function includes: The matrix operation unit is invoked, and based on the Mahalanobis distance calculation rule, and by introducing the inverse matrix of the feature space covariance matrix, the basic physical feature Mahalanobis distance between the physical feature vector of the unknown field and the reference vector of each standard field is calculated.

6. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 5, characterized in that, The steps of constructing and solving the global mapping loss function include: Combining the topological distance matrix and the preset standard field prior physical correlation coefficient matrix, a topological cross-penalty term for characterizing spatial association logic is calculated and generated; The global mapping loss function is constructed by combining and adding the Mahalanobis distance of the basic physical feature with the topological cross-penalty term.

7. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 6, characterized in that, The steps for solving the global mapping loss function include: The heuristic matching algorithm is invoked to iteratively calculate the global mapping loss function until the overall loss value reaches a preset convergence state. The set of mapping relationships that minimize the overall loss value is then searched and output as the mapping relationship.

8. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 1, characterized in that, The step of generating the streaming mask includes: When the heterogeneous data packet is determined to be a binary protocol packet with a fixed-length encoding mechanism, the fixed absolute byte offset of the target field in the memory block is calculated based on the mapping relationship and the data type byte length, and a static absolute displacement mask is generated as the streaming mask.

9. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 1, characterized in that, The step of generating the streaming mask includes: When the heterogeneous data packet is determined to be a text protocol packet with a variable length encoding mechanism, the structural symbol step vector required to traverse from the root node to the target address of the target field is extracted, and the single instruction stream multiple data stream extended instruction set is called to perform parallel scanning on the packet memory block to generate a single instruction stream multiple data stream topology projection mask as the streaming mask.

10. The method for standardized governance of multi-source heterogeneous data in city-level smart communities according to claim 1, characterized in that, When the comparison result of the structural topology fingerprint is consistent, according to the preset associated execution path in the historical fingerprint routing table, the heterogeneous data packet is controlled to directly flow to generate a streaming mask based on the mapping relationship and the network communication protocol type to which the heterogeneous data packet belongs, and then extract and store it.

Images