OT / IT heterogeneous data unified access and standardized conversion method

Abstract

The application relates to the technical field of big data analysis, and discloses an OT / IT heterogeneous data unified access and standardized conversion method, which comprises the following steps: acquiring a data stream and extracting service load data and metadata therefrom; converting the metadata into fixed-length hash features by using hash operation, stripping the metadata, and encapsulating the service load data and the hash features; dividing a plurality of continuous memory buffer areas corresponding to a bit mask base number of a memory space, and extracting feature offsets of the hash features by using the bit mask; sequentially appending and writing data into the buffer areas according to receiving time; and sequentially writing the data into a storage medium in batches when the memory occupation reaches a preset threshold. The application converts the device logical owner relationship into the continuity of the medium physical block by using the hash routing, suppresses write amplification, avoids system lagging caused by fragment consolidation, and guarantees the continuous and smooth data collection of a data surge.

Description

Technical Field

[0001] This invention belongs to the field of big data analytics technology, and in particular relates to a method for unified access and standardized conversion of heterogeneous OT / IT data. Background Technology

[0002] Currently, the mainstream technical approach is to integrate real-time production data and operational management data through a unified access platform. This involves using message middleware or buffer databases to perform heterogeneous protocol conversion, translating the collected raw information into a standardized data format to support subsequent fault prediction and statistical analysis. However, in scenarios with concurrent access from a massive number of data collection points, there is a significant difference between the production load data that is updated in seconds and the topology metadata that changes in a low frequency.

[0003] Existing solutions process data streams through full field mapping, resulting in frequent payload data repeatedly carrying static topology descriptions. This arrangement creates significant redundancy in physical storage space, leading to write amplification. While increasing storage hardware bandwidth can alleviate I / O pressure, performance bottlenecks caused by the mismatch between logical structure and physical storage characteristics persist unless the software control mechanism is reformed. For example, Chinese invention patent application CN119088803A discloses a data storage method, cloud storage system, and storage medium that converts data to be stored into data description text and determines the target storage server based on the hash value generated from the structured description text. To solve the above problems, simply increasing storage bandwidth or stacking hardware resources cannot eliminate the performance bottleneck caused by the mismatch between logical structure and physical storage characteristics. Multi-source data exhibits a discrete distribution when it reaches the storage medium, causing the memory table of the storage engine to be filled with disordered data blocks. When the system triggers a disk flush, the storage engine generates frequent merge storms, resulting in significant write latency under peak data surges.

[0004] Therefore, how to reconstruct the orchestration rules of heterogeneous data streams and achieve precise alignment between topological logic and the physical distribution of storage media has become the technical problem to be solved by this invention. Summary of the Invention

[0005] This invention aims to solve the problems of write amplification caused by metadata redundancy during the normalization process of heterogeneous data streams and the merge storm caused by massive concurrency in the storage engine.

[0006] This technical solution provides a method for unified access and standardized conversion of heterogeneous OT / IT data, comprising the following steps:

[0007] Step 101: Obtain heterogeneous data streams from the OT and IT ends, and extract business payload data and metadata representing the logical topology of the devices from the heterogeneous data streams.

[0008] Step 102: Perform a hash operation on the metadata to generate a fixed-length structural hash feature, strip the metadata, and encapsulate the business payload data and the structural hash feature into a standardized data block;

[0009] Step 103: In the memory space, multiple consecutive memory buffers corresponding to a preset bitmask base are pre-divided, and each consecutive memory buffer is assigned a unique memory address mapping key;

[0010] Step 104: Apply a preset bitmask to the structural hash features in the standardized data block, calculate the feature offset, and map the feature offset to the target mapping key;

[0011] Step 105: Based on the target mapping key, append the standardized data blocks to the contiguous memory buffer in the order of the received timestamps to map the logical layer device affiliation to the physical block continuity of the underlying storage medium.

[0012] Step 106: Monitor the memory water level of each continuous memory buffer. When the memory water level reaches the preset threshold, write the data blocks in the continuous memory buffer into the physical storage medium in a single batch.

[0013] Preferably, step 102, generating structural hash features and encapsulating standardized data blocks, includes the following steps: identifying multi-level association attributes in metadata and converting them into corresponding feature strings; using a hash algorithm to perform fixed-length compression mapping on the feature strings to calculate fixed-length structural hash features; establishing pointer associations between structural hash features and pre-stored relational dimension tables; and using pointer associations to exclude duplicate text descriptions in business payload data when storing standardized data blocks, in order to suppress data write amplification.

[0014] Preferably, step 103 involves dividing multiple contiguous memory buffers, including the following steps: step 401, obtaining the memory page size of the operating system; step 402, aligning the starting physical address of each contiguous memory buffer according to the memory page size, and setting the physical capacity of each contiguous memory buffer to an integer multiple of the memory page size.

[0015] Preferably, obtaining the heterogeneous data stream in step 101 includes the following steps: Step 501, receiving real-time sampling data from the energy production system in parallel through a multi-protocol adaptation layer; Step 502, performing timestamp normalization processing on the real-time sampling data to generate raw data with a unified message structure, wherein the sampling frequency of the real-time sampling data is not less than 1Hz.

[0016] Preferably, step 105, which appends the standardized data blocks to the continuous memory buffer in the order of the received timestamps, includes the following steps: step 601, identifying the high-frequency sampling payload in the standardized data blocks; step 602, allocating continuous offset addresses in the continuous memory buffer in the order of the received timestamps to complete the non-overwrite memory writing.

[0017] Preferably, the method further includes the following steps: Step 701, establishing a memory index table based on structural hash features; Step 702, upon receiving a data query instruction, retrieving the memory index table according to the structural hash features to be queried, and locating the corresponding contiguous memory buffer.

[0018] Preferably, step 106, which involves sequentially writing data blocks from the contiguous memory buffer into the physical storage medium in batches, includes the following steps: step 801, monitoring the I / O channel occupancy rate of the physical storage medium; step 802, initiating the batch data migration operation when the I / O channel occupancy rate is below 30% and the memory level of the contiguous memory buffer reaches a preset threshold.

[0019] Preferably, the encapsulation of standardized data blocks in step 102 includes the following steps: step 901, performing forward differential encoding on the business payload data to extract incremental change values; step 902, encapsulating the incremental change values ​​and structural hash features into a fixed-length standardized data block.

[0020] Preferably, the method further includes the following steps: Step 1001, constructing a storage block mapping table based on the target mapping key in the physical storage medium; Step 1002, pre-allocating contiguous storage block clusters according to the storage block mapping table, so that the storage space step size in the physical storage medium matches the physical capacity of the contiguous memory buffer in memory.

[0021] Compared with existing technologies, the OT / IT heterogeneous data unified access and standardized conversion method of the present invention has the following advantages:

[0022] 1. In the unified access of heterogeneous OT / IT data, by physically separating the topology metadata and production payload fields with different update frequencies, and using fixed-length topology hash fingerprints to replace text-based environment descriptions, the physical occupancy of a single time-series record is compressed from the underlying logic of the data structure. This eliminates the write amplification effect caused by repeated encapsulation of metadata in multi-source concurrent scenarios, ensuring that high-density real-time production information can be transmitted under limited network bandwidth resources. Through the dimensionality reduction and reorganization of the data structure, this mechanism enables the system to maintain a low storage space growth rate when facing hundreds of thousands of concurrent collection points, alleviating the addressing pressure on the core storage medium.

[0023] 2. By using the prefix routing mask of topological hash fingerprint, the logical device ownership relationship is directly mapped to the ring buffer address in physical memory. This allows discrete sensor sampling data to be physically clustered before reaching the disk medium, forcing the originally disordered discrete random writes into large block sequential append writes. This process effectively avoids write stuttering caused by frequent memory fragmentation when the storage engine is handling massive concurrent streams, bringing the disk write throughput close to the hardware physical limit and ensuring the continuity of processing when reporting surge data in industrial fields.

[0024] 3. By establishing a unique addressing pointer mapping between relational static dimension tables and time-series dynamic payload blocks through topological hash fingerprinting, the data retrieval process does not require cross-database and multi-table join queries. Data extraction across media can be achieved through a single hash index. This mechanism improves query response latency from logical-level search optimization to direct reading at the physical memory pointer level. While ensuring the consistency between IT management data and OT production data, it shortens the response cycle of the data platform when pulling massive historical records on demand. Attached Figure Description

[0025] Figure 1 This is a flowchart of the unified access and standardized disk-based processing of heterogeneous OT / IT data in this invention;

[0026] Figure 2 This is a diagram of metadata structure hash extraction and standardized data block encapsulation in this invention. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0028] It should be noted that all directional and positional terms used in this invention, such as: up, down, left, right, front, back, vertical, horizontal, inner, outer, top, bottom, transverse, longitudinal, center, etc., are only used to explain the relative positional relationship and connection between components in a specific state (as shown in the accompanying drawings). They are only for the convenience of describing this invention and do not require that this invention be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention. In addition, the descriptions of "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.

[0029] In the description of this invention, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections; they can refer to direct connections or indirect connections through an intermediate medium; they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0030] In the description of this specification, references to the terms "an embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example, and the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0031] A method for unified access and standardized transformation of heterogeneous OT / IT data includes the following steps:

[0032] Step 101: Obtain heterogeneous data streams from the OT and IT ends, and extract business payload data and metadata representing the logical topology of the devices from the heterogeneous data streams.

[0033] Step 102: Perform a hash operation on the metadata to generate a fixed-length structural hash feature, strip the metadata, and encapsulate the business payload data and the structural hash feature into a standardized data block;

[0034] Step 103: In the memory space, multiple consecutive memory buffers corresponding to a preset bitmask base are pre-divided, and each consecutive memory buffer is assigned a unique memory address mapping key;

[0035] Step 104: Apply a preset bitmask to the structural hash features in the standardized data block, calculate the feature offset, and map the feature offset to the target mapping key;

[0036] Step 105: Based on the target mapping key, append the standardized data blocks to the contiguous memory buffer in the order of the received timestamps to map the logical layer device affiliation to the physical block continuity of the underlying storage medium.

[0037] Step 106: Monitor the memory water level of each continuous memory buffer. When the memory water level reaches the preset threshold, write the data blocks in the continuous memory buffer into the physical storage medium in a single batch.

[0038] Preferably, step 102, generating structural hash features and encapsulating standardized data blocks, includes the following steps: identifying multi-level association attributes in metadata and converting them into corresponding feature strings; using a hash algorithm to perform fixed-length compression mapping on the feature strings to calculate fixed-length structural hash features; establishing pointer associations between structural hash features and pre-stored relational dimension tables; and using pointer associations to exclude duplicate text descriptions in business payload data when storing standardized data blocks, in order to suppress data write amplification.

[0039] Preferably, the feature offset is extracted in step 104 according to the following calculation rule: P=H&M, where P is the calculated feature offset, H is the structural hash feature in the standardized data block, and M is the preset bitmask.

[0040] Preferably, step 103 involves dividing multiple contiguous memory buffers, including the following steps: step 401, obtaining the memory page size of the operating system; step 402, aligning the starting physical address of each contiguous memory buffer according to the memory page size, and setting the physical capacity of each contiguous memory buffer to an integer multiple of the memory page size.

[0041] Preferably, obtaining the heterogeneous data stream in step 101 includes the following steps: Step 501, receiving real-time sampling data from the energy production system in parallel through a multi-protocol adaptation layer; Step 502, performing timestamp normalization processing on the real-time sampling data to generate raw data with a unified message structure, wherein the sampling frequency of the real-time sampling data is not less than 1Hz.

[0042] Preferably, step 105, which appends the standardized data blocks to the continuous memory buffer in the order of the received timestamps, includes the following steps: step 601, identifying the high-frequency sampling payload in the standardized data blocks; step 602, allocating continuous offset addresses in the continuous memory buffer in the order of the received timestamps to complete the non-overwrite memory writing.

[0043] Preferably, the method further includes the following steps: Step 701, establishing a memory index table based on structural hash features; Step 702, upon receiving a data query instruction, retrieving the memory index table according to the structural hash features to be queried, and locating the corresponding contiguous memory buffer.

[0044] Preferably, step 106, which involves sequentially writing data blocks from the contiguous memory buffer into the physical storage medium in batches, includes the following steps: step 801, monitoring the I / O channel occupancy rate of the physical storage medium; step 802, initiating the batch data migration operation when the I / O channel occupancy rate is below 30% and the memory level of the contiguous memory buffer reaches a preset threshold.

[0045] Preferably, the encapsulation of standardized data blocks in step 102 includes the following steps: step 901, performing forward differential encoding on the business payload data to extract incremental change values; step 902, encapsulating the incremental change values ​​and structural hash features into a fixed-length standardized data block.

[0046] Preferably, the method further includes the following steps: Step 1001, constructing a storage block mapping table based on the target mapping key in the physical storage medium; Step 1002, pre-allocating contiguous storage block clusters according to the storage block mapping table, so that the storage space step size in the physical storage medium matches the physical capacity of the contiguous memory buffer in memory.

[0047] Example 1: In the continuous operation environment of a large energy base, the system faces the challenge of processing concurrent data streams from hundreds of thousands of sensor nodes. This base aggregates high-frequency sampling loads at the second level generated by wind and photovoltaic equipment, as well as multi-level equipment topology relationships in the information technology management system. Under a conventional big data fusion architecture, the synchronous parsing of map metadata and high-frequency loads causes the frequently occurring load data to repeatedly carry static topology descriptions. When heterogeneous data is concurrently written to disk, massive discrete time-series records in the memory table of the time-series database trigger write amplification and fragmentation merging storms, causing the physical storage medium to face pressure on input / output channel occupancy and write latency. The system receives energy production data in parallel through a multi-protocol adaptation layer. The system uses real-time sampling data with a sampling frequency of no less than 1Hz to extract service payload data and metadata representing the logical topology of devices from heterogeneous data streams. The metadata is hashed to generate fixed-length structural hash features. The metadata is stripped and the service payload data and structural hash features are encapsulated into standardized data blocks. At the same time, the memory page size of the operating system is obtained in the memory space. The starting physical address of each continuous memory buffer is aligned according to the memory page size, and the physical capacity of each continuous memory buffer is set to an integer multiple of the memory page size. In this way, multiple continuous memory buffers corresponding to a preset bitmask base are pre-divided, and a unique memory address mapping key is assigned to each continuous memory buffer.

[0048] The processing unit applies a preset bitmask to the structural hash features in the standardized data block and calculates the feature offset according to the calculation rule P=H&M, where P is the calculated feature offset, H is the structural hash feature in the standardized data block, and M is the preset bitmask. The processing unit maps the feature offset to a target mapping key, identifies the high-frequency sampling payload in the standardized data block based on the target mapping key, and allocates consecutive offset addresses in the corresponding contiguous memory buffer according to the ascending order of the received timestamp, completing non-overwrite memory writing. This maps the logical-level device affiliation to the physical block continuity of the underlying storage medium. The storage controller establishes a memory index based on the structural hash features. The table monitors the memory level of each contiguous memory buffer and the I / O channel occupancy of the physical storage medium. When the I / O channel occupancy is below 30% and the memory level of the contiguous memory buffer reaches a preset threshold, a batch data migration operation is initiated. This operation sequentially writes data blocks from the contiguous memory buffer to the physical storage medium in a single batch. This hash-based memory sharding routing and alignment append mechanism converts the random discrete writing of massive discrete sensor sampling data into sequential append writing. This avoids the write amplification effect caused by repeated encapsulation of metadata in multi-source concurrent scenarios, suppresses the merge storm caused by memory fragmentation in the storage engine, and maintains the system stability during the industrial real-time data aggregation process.

[0049] Example 2: When evaluating the performance of multi-source concurrent access physical storage, a test platform was constructed using a standardized test server equipped with non-volatile storage media and a peak random write speed of 500MB / s. An open-source industrial control system dataset was imported as the data source. This data source continuously injected time-series samples containing device logical topology relationships at a frequency of 100Hz. Simultaneously, 15% of redundant network retransmission load and timestamp out-of-order disturbances were mixed into the data link to simulate network congestion in an industrial setting, placing the test platform in a high-concurrency discrete data stream input environment containing duplicate metadata. This was done for continuous memory buffers. The memory waterline setting is based on the physical constraints of the size of consecutive data blocks written to disk in a single operation and the flushing frequency of the internal cache of the physical storage medium. When the memory waterline setting is below the limit, the disk write operation is too frequent, which makes it impossible to fully activate the sequential write attribute of the non-volatile storage medium. When the setting is above the limit, the memory reclamation delay will cause the risk of memory overflow. By establishing a mapping model of memory residence period and I / O channel utilization, the working range of the memory waterline is determined to be 80% to 90% of the total capacity of the continuous memory buffer. This range maintains the physical balance between memory consumption and disk throughput.

[0050] The system was configured in parallel with three control groups: a control group 1 using traditional cross-table joint writes, a control group 2 using structured hash routing but without memory page alignment, and experimental groups 1, 2, and 3 of this invention with memory watermarks set at 60%, 85%, and 98%, respectively. After a continuous 30-minute high-frequency load impact, underlying performance monitoring indicators were extracted. Control group 1, which did not remove duplicate metadata, experienced fragmentation and merging, causing the average write latency to climb to 45.8ms and the I / O channel occupancy rate to remain at a full load of 96.2%. Control group 2 eliminated some write amplification effects, reducing latency to 18.4ms. Experimental group 1 of this invention exhibited a write latency of 12.3ms. Experimental group 2 of this invention relied on a continuous memory buffer matching the operating system's memory page size and a batch data migration operation triggered by an 85% memory watermark. Its processing unit extracted metadata and calculated feature bias according to the calculation rule P=H&M. The method uses a displacement parameter, where P is the feature offset, H is the structural hash feature, and M is a preset bitmask. Based on the target mapping key, the service payload data is clustered and appended according to the received timestamp order, achieving a disk throughput of 312.6 MB / s and a stable write latency of 5.4 ms. In contrast, in test group three, where the memory waterline is set to 98%, the write latency rebounded to 22.7 ms due to frequent page swapping triggered by the operating system near the physical memory limit, exhibiting a performance inflection point consistent with the underlying storage physical laws. Combined with the physical mapping results of the aforementioned multi-dimensional gradient data, memory sharding routing and aligned append operations based on structural hash features reduce the write amplification effect caused by concurrent disk writes of high-frequency discrete time-series data. This method clusters and stores service payload data according to the received timestamp order and converts the logical topology affiliation into the physical block continuity of the underlying storage medium, improving the data lake write throughput and system stability during the unified access process for heterogeneous data.

[0051] Example 3: In the continuous operation environment of a large energy base, the multi-protocol adaptation layer obtains metadata containing hierarchical topology relationships from heterogeneous devices. Unstructured plant, equipment, and measurement point hierarchical information faces limitations such as topology hierarchical order disorder and hash collisions when converted into hash features. The preset bitmask deviates from the cardinality constraint of the physical memory buffer, causing the memory address mapping key to exceed the addressing boundary of the continuous memory buffer, leading to memory out-of-bounds access and data aggregation stagnation. The processing unit extracts the hierarchical identifiers representing the logical topology relationships of the devices from the metadata, arranges the hierarchical identifiers in lexicographical order from high to low hierarchical depth, inserts underscore separators between adjacent hierarchical identifiers, and combines them to form a feature string. The processing unit uses a secure hash algorithm to calculate the feature string and outputs a fixed-length 256-bit structured hash feature. Based on the principle of binary addressing boundary alignment at the computer's underlying level, the system enforces physical sharding cardinality. The constraint procedure involves the storage controller extracting the total physical memory capacity allocated by the system to the current process and the operating system's base memory page size during the pre-division of contiguous memory buffers. The quotient of the total physical memory capacity and the base memory page size is calculated to determine the theoretical maximum number of fragments. The system rounds down to select the largest power of 2 not greater than the theoretical maximum number of fragments as the boundary constraint condition and obtains the total number of contiguous memory buffers, defined as variable N. During the pre-division of multiple contiguous memory buffers, the storage controller obtains the total number of contiguous memory buffers and determines the effective bit length of the mask according to the formula L=log2N, where L is the effective bit length of the mask and N is the total number of contiguous memory buffers. The storage controller generates a sequence where the lower L bits are binary 1s and the remaining higher bits are binary 0s as a preset bitmask. The processing unit calculates the bitwise AND operation result of the structural hash feature and the preset bitmask to determine the feature offset.

[0052] To prevent hash local clustering caused by high-frequency sampling at specific equipment points during long-term operation, the parallel execution process of the system uses a quantization procedure. The storage controller continuously extracts the current physical byte backlog of each continuous memory buffer according to a sampling period synchronized with the received timestamp accuracy. It calculates the real-time variance of the backlog, including all continuous memory buffers, based on the standard variance formula. The sliding window for variance calculation is set to the sampling data within the past 300 seconds, and the full buffer backlog is polled every 1000ms. The steady-state extreme value threshold is set to 20% of the current average backlog of all buffers. If the instantaneously calculated variance value exceeds this threshold three times consecutively, hash local clustering is determined to have occurred. When the real-time variance exceeds the preset steady-state extreme value threshold, the storage controller... The storage controller triggers a forced load balancing mechanism, extracting the standardized data blocks that exceed the limits in the highest-order continuous memory buffer of the physical byte backlog, and sequentially appending them to the lowest-order candidate continuous memory buffer until the real-time dispersion variance falls back to the steady-state extreme value threshold range, smoothing out the fragmentation loss of physical media under long-term operation; the splicing of hierarchical identifiers and the calculation of bitmasks construct a mapping path from unstructured logical topology to fixed-length physical addressing keys. The aforementioned operations constrain discrete business load data at the same measurement point level within the specified feature offset addressing boundary at the physical memory level, eliminating hash collisions and memory out-of-bounds phenomena in the process of unified access of heterogeneous data, and maintaining the high-frequency write stability and data throughput of the industrial real-time data lake.

[0053] Example 4: When the system is deployed to a heterogeneous physical storage hardware environment for the first time, the storage controller initiates a pre-deployment calibration operation before accessing actual production data to determine the preset threshold of the memory water level that is compatible with the current hardware performance. The storage controller injects a simulated test data stream with a preset frequency into the multi-protocol adaptation layer. This stream contains structural features and timestamps that conform to the topology depth of the target device. The processing unit continuously writes the simulated test data stream into a continuous memory buffer according to the structural hash sharding routing mechanism. During the writing process, the storage controller extracts the real-time memory fill rate of the continuous memory buffer and simultaneously initiates a sequential write operation request with a fixed block size to the underlying physical storage medium. It also collects the peak sequential write throughput when the physical storage medium continuously receives data blocks and the inherent flushing delay time of a single physical disk write.

[0054] After obtaining the physical measurement parameters, the storage controller calculates them according to the formula. Determine the preset threshold for the memory water level, where, To preset a threshold for the determined memory water level, This represents the total physical capacity of the contiguous memory buffer. The inherent flushing delay time for a single physical disk placement. To determine the real-time memory fill rate of the continuous memory buffer, the storage controller stores the predetermined memory waterline threshold in the memory index table. To offset the calculation error caused by sampling frequency fluctuations, the real-time memory fill rate of the continuous memory buffer is smoothed using an exponentially weighted moving average algorithm, with a sampling step size of 100ms and a smoothing coefficient of 0.8. When calculating the predetermined memory waterline threshold, an additional safety margin coefficient of 1.2 is introduced to ensure that even if there is a sudden 20% surge in traffic within the inherent flushing delay time of physical disk disposal, the buffer still has sufficient physical remaining space to accommodate the data to be processed. This serves as the triggering benchmark for subsequent batch data migration operations. The pre-calibration operation calculates the overflow prevention boundary based on the physical hardware parameters of the target environment to adapt to the differences in media read / write performance during cross-platform migration.

[0055] Example 5: When the system faces a new batch of physical storage media deployment, the storage controller initiates an offline baseline calibration operation of the underlying hardware before the multi-protocol adaptation layer accesses the service network to determine the inherent flushing delay time of a single physical disk write. Total physical capacity of contiguous memory buffer To determine the adaptation boundary, the storage controller continuously injects pseudo-random test sequences that match the topology depth of the target device into the physical storage medium. The write load is increased incrementally according to the system bus bandwidth threshold. Simultaneously, the controller monitors the instruction queuing depth and page erase latency of the storage medium's main control chip. The storage controller identifies the physical critical point where the instruction queuing depth reaches saturation and the page erase latency experiences a sudden data change. The single write time corresponding to this physical critical point is extracted as the inherent flushing delay time for a single physical disk write operation. The calibration value is used as a benchmark to calculate the total physical capacity of the contiguous memory buffer according to a preset ratio. .

[0056] After obtaining the calibration parameters, the storage controller defines continuous physical address boundaries within the physical memory space according to the operating system's memory page size, and calculates the total physical capacity. Divide into multiple aligned contiguous memory buffers and reduce the inherent flushing delay of a single physical disk push. The control register written to the system kernel serves as a baseline parameter. The aforementioned offline baseline calibration operation provides a low-level hardware parameter baseline for calculating the preset threshold of the memory water level by measuring the physical feedback of the physical storage hardware under load impact. This avoids memory overflow defects caused by differences in the scheduling of control chips in storage media from different manufacturing batches, and maintains the system operation stability and data write throughput of the heterogeneous data unified access architecture during cross-platform migration and deployment.

[0057] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit of this application and the scope of protection of this invention, and all of these forms are within the protection scope of this application.

Claims

1. A method for unified access and standardized conversion of heterogeneous OT / IT data, characterized in that, Includes the following steps: Step 101: Obtain heterogeneous data streams from the OT and IT ends, and extract business payload data and metadata representing the logical topology of the devices from the heterogeneous data streams. Step 102: Perform a hash operation on the metadata to generate a fixed-length structural hash feature, strip the metadata, and encapsulate the business payload data and the structural hash feature into a standardized data block; Step 103: In the memory space, multiple consecutive memory buffers corresponding to a preset bitmask base are pre-divided, and each consecutive memory buffer is assigned a unique memory address mapping key; Step 104: Apply a preset bitmask to the structural hash features in the standardized data block, calculate the feature offset, and map the feature offset to the target mapping key; Step 105: Based on the target mapping key, append the standardized data blocks to the contiguous memory buffer in the order of the received timestamps to map the logical layer device affiliation to the physical block continuity of the underlying storage medium. Step 106: Monitor the memory water level of each continuous memory buffer. When the memory water level reaches the preset threshold, write the data blocks in the continuous memory buffer into the physical storage medium in a single batch.

2. The method for unified access and standardized conversion of OT / IT heterogeneous data according to claim 1, characterized in that, Step 102 generates a structured hash feature and encapsulates standardized data blocks, including the following steps: identifying multi-level association attributes in metadata and converting them into corresponding feature strings; using a hash algorithm to perform fixed-length compression mapping on the feature strings to calculate fixed-length structured hash features; establishing pointer associations between the structured hash features and pre-stored relational dimension tables; and using pointer associations to exclude duplicate text descriptions in business payload data when storing standardized data blocks, in order to suppress data write amplification.

3. The method for unified access and standardized conversion of OT / IT heterogeneous data according to claim 1, characterized in that, Step 103 involves dividing the memory into multiple contiguous buffers, including the following steps: Step 401, obtaining the memory page size of the operating system; Step 402, aligning the starting physical address of each contiguous memory buffer according to the memory page size, and setting the physical capacity of each contiguous memory buffer to an integer multiple of the memory page size.

4. The method for unified access and standardized conversion of OT / IT heterogeneous data according to claim 1, characterized in that, Step 101 involves acquiring heterogeneous data streams, including the following steps: Step 501, receiving real-time sampling data from the energy production system in parallel through a multi-protocol adaptation layer; Step 502, performing timestamp normalization on the real-time sampling data to generate raw data with a unified message structure, wherein the sampling frequency of the real-time sampling data is not less than 1Hz.

5. The method for unified access and standardized conversion of OT / IT heterogeneous data according to claim 1, characterized in that, Step 105 involves appending standardized data blocks to a contiguous memory buffer in the order of received timestamps, including the following steps: Step 601, identifying high-frequency sampling payloads in the standardized data blocks; Step 602, allocating consecutive offset addresses within the contiguous memory buffer in ascending order of received timestamps to complete non-overwrite memory writing.

6. The method for unified access and standardized conversion of OT / IT heterogeneous data according to claim 1, characterized in that, It also includes the following steps: Step 701: Establish a memory index table based on structural hash features; Step 702: Upon receiving a data query instruction, retrieve the memory index table based on the structure hash feature to be queried, and locate the corresponding contiguous memory buffer.

7. The method for unified access and standardized conversion of OT / IT heterogeneous data according to claim 1, characterized in that, Step 106 involves sequentially writing data blocks from the contiguous memory buffer to the physical storage medium in batches, including the following steps: Step 801, monitoring the I / O channel occupancy rate of the physical storage medium; Step 802, initiating the batch data migration operation when the I / O channel occupancy rate is below 30% and the memory level of the contiguous memory buffer reaches a preset threshold.

8. The method for unified access and standardized conversion of OT / IT heterogeneous data according to claim 1, characterized in that, Step 102 encapsulates standardized data blocks, including the following steps: Step 901, performing forward differential encoding on the business payload data to extract incremental change values; Step 902, encapsulating the incremental change values ​​and structural hash features into fixed-length standardized data blocks.

9. The method for unified access and standardized conversion of OT / IT heterogeneous data according to claim 1, characterized in that, It also includes the following steps: Step 1001: Construct a storage block mapping table based on the target mapping key in the physical storage medium; Step 1002: Pre-allocate contiguous storage block clusters according to the storage block mapping table so that the storage space step size in the physical storage medium matches the physical capacity of the contiguous memory buffer in memory.

Images