Intelligent data storage method for smart energy unit

Abstract

The application discloses an intelligent data storage method for a smart energy unit, comprising: analyzing the sampling characteristics of an original data stream to generate a frequency characteristic vector set; based on the frequency characteristic vector set, performing adaptive hierarchical routing and constructing a virtual time grid to generate a hierarchical data queue and a virtual alignment index; for the hierarchical data queue, performing elastic buffering and out-of-order reorganization to output an ordered data block and a position mapping table; and parallel persisting each layer of the ordered data block, fusing the position mapping table and the virtual alignment index, and constructing a multi-dimensional index tree containing frequency, time and position dimensions. The application can optimize the configuration of storage resources and adaptively adjust the processing flow according to the internal sampling frequency characteristics of data, thereby improving the storage density and query analysis efficiency of massive heterogeneous energy data.

Description

Technical Field

[0001] This invention belongs to the field of energy Internet of Things data processing, and in particular, it is an intelligent data storage method for smart energy units. Background Technology

[0002] With the widespread deployment of various terminal devices such as smart meters, photovoltaic inverters, and energy storage units, the data collected by the system exhibits unprecedented complexity: data sources comply with multiple communication protocols such as DL / T645 and IEC104; data sampling frequencies range from minute-level environmental parameters to millisecond- and even microsecond-level power quality monitoring, spanning several orders of magnitude in time scale; and the total amount of data is growing exponentially. This data environment poses a severe challenge to traditional storage systems. The common device + time storage model leads to a large amount of metadata redundancy and low storage space utilization. More importantly, the fixed storage architecture cannot adapt to the inherent time scale differences in data, resulting in conflicts between the intensive writing of high-frequency data and the sparse storage of low-frequency data. This not only exacerbates the wear and tear of storage media such as SSDs but also makes cross-frequency and cross-time scale correlation queries and analysis operations extremely inefficient, with latency often increasing by hundreds of milliseconds, thus restricting further data value extraction. Therefore, researching and designing a new intelligent data storage method that can sense and adapt to the time scale characteristics of data has significant theoretical and engineering implications for improving the overall performance of smart energy systems and supporting advanced upper-level applications.

[0003] Currently, to address the dynamic nature of energy data, several intelligent storage management solutions have been proposed. One representative technology is a dynamic storage structure migration method based on activity scoring. This method aims to solve the problem of dynamic additions and subtractions of data collection items in the system. Specifically, this method calculates a dynamic activity score for each reported data item, which integrates factors such as the frequency of occurrence of the data item within a time window, the magnitude of data value changes, and data quality. The system uses this activity score to determine the importance of a data item. When a newly emerging data item's activity score consistently exceeds a preset promotion threshold, the system determines it as a stable and important data item and triggers a background storage structure change, such as dynamically adding a dedicated field for it in the wide table of the main storage and migrating the data item's historical data there. Conversely, when an existing data item is inactive for a long time and its vacancy rate consistently exceeds a degradation threshold, the system will demote it from the expensive main storage area and migrate it to a cold storage medium with a higher compression ratio to free up storage resources. To make the decision-making process for promotion and demotion smoother and avoid frequent system changes due to minor fluctuations in scores, this method also introduces a fuzzy logic control model to set a transition range for the decision threshold.

[0004] While the aforementioned activity-based scoring schemes have achieved some degree of dynamic and intelligent data storage, their core idea—judging the presence and activity of data items—is essentially a one-size-fits-all management approach. It ignores the more fundamental and crucial intrinsic dimension of energy data—the time scale, i.e., the significant differences in sampling frequencies. This neglect leads to a series of deep-seated technical problems when handling multi-rate heterogeneous data streams, primarily manifested in resource mismatch in out-of-order buffers, storage and indexing conflicts for multi-rate data, and loss of time alignment accuracy during cross-frequency analysis. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent data storage method for smart energy units, in order to solve the aforementioned problems existing in the prior art.

[0006] The technical solution, a method for intelligent data storage in smart energy units, includes:

[0007] Analyze the sampling characteristics of the raw data stream from the smart energy unit to generate a set of frequency feature vectors;

[0008] Adaptive hierarchical routing is performed based on the frequency feature vector set, and a virtual time grid is constructed to generate a hierarchical data queue and a virtual alignment index.

[0009] Elastic buffering and out-of-order reordering are performed on the hierarchical data queue to produce ordered data blocks and a location mapping table;

[0010] The ordered data blocks are persisted in parallel, and the location mapping table and virtual alignment index are merged to build a multi-dimensional index tree that includes frequency, time and location dimensions.

[0011] Beneficial effects: This invention can optimize the configuration of storage resources and adaptively adjust the processing flow based on the inherent sampling frequency characteristics of the data, thereby improving the storage density and query and analysis efficiency of massive heterogeneous energy data. Attached Figure Description

[0012] Figure 1 A flowchart illustrating the steps of an intelligent data storage method for a smart energy unit, as provided in this application embodiment.

[0013] Figure 2 A flowchart illustrating the steps for generating a hierarchical data queue, as provided in an embodiment of this application.

[0014] Figure 3 A flowchart illustrating the steps for generating a virtual alignment index provided in this application embodiment.

[0015] Figure 4 A flowchart illustrating the steps for creating a discrete virtual time axis, as provided in an embodiment of this application. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0018] The study found that out-of-order data buffering strategies lack frequency awareness, leading to low buffering efficiency and resource mismatch. Existing technologies typically use a uniform buffer window to handle out-of-order data arriving due to network latency. However, for a high-frequency data stream with a sampling rate of 1kHz, a 50ms network jitter means that 50 data points need to be buffered and rearranged; while for a low-frequency data stream of 1Hz, the same 50ms jitter hardly constitutes an out-of-order problem. A uniform buffer window is too large for low-frequency data, causing unnecessary memory consumption and data processing latency; while for high-frequency data, it may be too small to accommodate its out-of-order scale, causing a large amount of data to be prematurely flushed into an inefficient overflow area, increasing the computational overhead of subsequent reassembly. Secondly, the storage layout and indexing mechanism do not distinguish between data temporal density, causing serious read / write performance conflicts. Because data is not physically isolated according to sampling frequency, high-frequency dense data points and low-frequency sparse data points are forced to be mixed and stored in the same logical data block. The continuous writing of high-frequency data triggers constant read-modify-write operations and defragmentation of the entire data block, severely interfering with stable access to low-frequency data within the same region. At the indexing level, commonly used time-series indexes such as B+ trees become highly unbalanced under mixed-density data. Index nodes frequently split or merge due to significant differences in fill rates, resulting in suboptimal performance for both point queries on high-frequency data and range scans on low-frequency data. Finally, the lack of a unified multi-resolution time base makes it difficult to guarantee the accuracy and efficiency of cross-frequency data correlation analysis. When it is necessary to analyze the relationship between high-frequency voltage and low-frequency temperature changes, existing technologies, lacking a unified time alignment mechanism at the underlying level, can only perform expensive, ad-hoc resampling or interpolation calculations during queries. This post-processing approach not only increases query latency but also makes the choice of interpolation algorithms difficult to universally apply, easily introducing large errors and failing to meet the precise time alignment requirements of many advanced analytical applications (such as fault diagnosis and supply and demand forecasting).

[0019] In the field of smart energy, early data storage methods attempted to address the problem of dynamic additions and subtractions of data items. For example, one approach was to establish a storage structure migration mechanism based on spatiotemporal coupling dynamic weights, enabling autonomous conversion between horizontal and vertical table hybrid storage. Specifically, the activity of newly emerging data items is first assessed. For instance, when the system detects a new data item (such as off-grid frequency, data tag 0x813) being reported, a gated event awareness scoring model (GTAF) is used to calculate its initial activity score. This activity score S(t) can be calculated using the formula S(t) = α·f(t) + β·ΔQ; where S(t) is the activity score at time t; α is the time decay coefficient, used to reflect that more recent data has higher value; f(t) is the frequency of the data item within the time window; β is the data quality weight, used to measure the validity and completeness of the data; and ΔQ is the magnitude of the data value change. When the score consistently exceeds a preset threshold (e.g., an average of 0.89 over three consecutive collection periods, exceeding the promotion threshold of 0.85), the system triggers a storage upgrade, dynamically adding a field (e.g., off_frequency) to the existing table and migrating the historical data for that data item from the temporary storage area to the new field. Correspondingly, a degradation mechanism is also included. The system periodically checks the vacancy rate V(t) of each data item, calculated as V(t) = 1 - (number of valid records / total number of records); where V(t) is the vacancy rate; the number of valid records is the number of records that actually store valid values ​​within a specified time period; and the total number of records is the theoretically expected total number of records within that time period. When the vacancy rate of a non-core parameter (e.g., ambient temperature) consistently exceeds a threshold (e.g., V(t) > 0.6), the system classifies it as low-activity data and migrates it from the main storage area to a cold storage area with a higher compression ratio to free up valuable main storage space. It can be seen that this scheme uses activity to determine the importance of data and dynamically adjusts the table structure. However, this solution primarily focuses on the frequency of data item occurrence and vacancy rate, without delving into the inherent sampling frequency characteristics of the data stream. In scenarios with heterogeneous energy terminals, the sampling rate of the data stream can range from once per minute to thousands of times per second. Storing high-sampling-rate (high-frequency) data and low-sampling-rate (low-frequency) data in the same table using the same strategy leads to frequent writes of high-frequency data, resulting in significant storage fragmentation and index maintenance overhead, while low-frequency data generates numerous invalid gaps, wasting storage space. Furthermore, it fails to provide targeted optimization when querying data of different frequencies, resulting in poor query efficiency.

[0020] like Figure 1 As shown, an intelligent data storage method for smart energy units is proposed, including the following steps:

[0021] The sampling characteristics of the raw data stream originating from the smart energy unit are analyzed to generate a set of frequency feature vectors.

[0022] In this embodiment, the raw data stream refers to the raw byte stream accessed from various energy terminals (such as smart meters, inverters, energy storage units, etc.) via different communication protocols such as DL / T645 and IEC104. Specifically, these byte streams need to undergo multi-protocol parsing to extract data item identifiers, timestamps, and values, forming standardized data records. A sliding time window is established for each data item, for example, a window of length T. w = 1000ms window. The number of data points N appearing within each window is counted. i Calculate the instantaneous sampling frequency f of the data item. i = N i / T w Based on the real-time sampling frequency sequence obtained from multiple consecutive windows, its statistical characteristics are further calculated, and finally a frequency feature vector is constructed for each data item. This vector is a triple (f avg f max S f ); where f avg The frequency mean, i.e., μ f ;f max S represents the maximum frequency. f For frequency stability, the formula is S. f =1-σ f / μ f , σ f denoted as the standard deviation of the frequency. This vector set comprehensively characterizes the sampling characteristics of each data stream from both time and volatility dimensions.

[0023] Adaptive hierarchical routing is performed based on the frequency feature vector set, and a virtual time grid is constructed to generate a hierarchical data queue and a virtual alignment index.

[0024] In this embodiment, based on the frequency characteristics of the data, they are categorized and sent to different processing channels, and a unified time benchmark is established for subsequent cross-frequency data alignment. Specifically, the system uses clustering algorithms such as K-means to analyze the frequency feature vectors of all data items, dynamically dividing the data into three levels: high frequency, medium frequency, and low frequency, and generating dynamic thresholds between levels. Based on the frequency feature vector and dynamic threshold of each data item, it is routed to the corresponding hierarchical data queue. Simultaneously, the system uses a globally minimal time granularity Δt... min Using 1ms as a baseline, construct a discrete, unified virtual time axis VT = {t0, t0 + Δt}. min , t0 + 2Δt min ...}. The actual timestamp t of each data point that actually arrives. real They will all pass through a mapping function g(t)real ) =└ (t real - t0) / Δt min ┘, a unique virtual grid position v mapped onto the virtual timeline. pos Where └ ┘ represents the floor function, and t0 is the start time of the time axis. The final generated virtual alignment index I... align It records the correspondence between each data point and its virtual location and the level it belongs to, in the form of (data id v pos layer id ), where data id As a unique identifier for data points, layer id The frequency level to which the data point belongs (e.g., high frequency, medium frequency, or low frequency).

[0025] Elastic buffering and out-of-order reordering are performed on the hierarchical data queue to produce ordered data blocks and a location mapping table.

[0026] Specifically, due to factors such as network latency, the order in which data arrives at the computing node may not match the order of its generated timestamps, i.e., out-of-order. This implementation can efficiently handle out-of-order data. The system establishes an elastic buffer for each hierarchical data queue. Optionally, the size of this buffer, i.e., the elastic time window W(t), is not fixed, but dynamically adjusted according to the regularity of data arrival. Its adjustment formula is W(t) = W base ·(1 +α·σ(Δt) / μ(Δt)); where W base α is the baseline window size; σ(Δt) and μ(Δt) are the standard deviation and mean of the data arrival time interval, respectively. The more irregular the data arrival (i.e., the larger the ratio of σ(Δt) / μ(Δt)), the larger the window, which can accommodate and wait for more potentially late data. Within the window, the system reserves storage space for upcoming data and manages out-of-order data that actually arrives, for example, by writing it to a temporary overflow area. When the window trigger condition is met (such as when the time is reached or the data volume reaches a threshold), the system performs a merge sort on the data in the main buffer and the overflow area, generating a data block with completely ordered timestamps. Simultaneously, a location mapping table M is generated. pos It records the mapping relationship between the logical location of each data point and its physical location in the ordered data block.

[0027] The ordered data blocks are persisted in parallel, and the location mapping table and virtual alignment index are merged to build a multi-dimensional index tree that includes frequency, time and location dimensions.

[0028] Specifically, data is written to physical storage and the final index is built. The system writes ordered data blocks from different levels to physical storage media (such as SSDs) in parallel and asynchronously through independent I / O channels. During or after the write operation, the system merges two key metadata sets: a virtual alignment index I... align and location mapping table M pos Based on this information, the system extracts a three-dimensional coordinate for each data point. This coordinate includes time information, frequency hierarchy information, and physical storage location information. Based on these three-dimensional coordinates, an efficient multi-dimensional index structure, such as a B+ tree, is constructed. The leaf nodes of this index tree store data in the form of (v... time layer id , physical addr The triplet of ) allows the system to quickly locate data based on any one or more dimensions of time, frequency, and location, where v time For indexing the position of data points on the virtual timeline, physical addr This refers to the actual storage address or offset of the data point in the physical storage medium.

[0029] This embodiment can achieve data layering, buffering, reorganization, persistence, and indexing based on the inherent characteristics of data sampling frequency, thereby achieving a better balance between storage space efficiency and query performance.

[0030] In a specific application scenario, a smart energy system may simultaneously connect to hundreds or even thousands of data measurement points, with sampling frequencies ranging from ambient temperature every few minutes to power quality monitoring thousands of times per second—a vast difference. Using a uniform storage and processing strategy would lead to system inefficiency. Therefore, this embodiment provides an adaptive data flow navigation mechanism. Specifically, as... Figure 2 As shown, a hierarchical data queue is generated, including:

[0031] The K-means clustering algorithm is used to process the frequency feature vector set, and the dynamic stratification threshold is calculated.

[0032] Specifically, the system generates a frequency mean f for each data item. avg Maximum frequency f max and frequency stability S f The frequency feature vector set refers to the collection of frequency feature vectors of all active data items in the current system. To obtain a dynamic stratification threshold, preferably, instead of using fixed frequency values ​​(e.g., hardcoding >100Hz as high frequency), a K-means clustering algorithm that is more adaptable to changes in the overall data distribution is employed. In one possible implementation: the mean frequency f of all data items is extracted from the frequency feature vector set. avgAs input data for clustering, the selection of initial cluster centers is crucial for improving the stability and convergence speed of the clustering algorithm. In a preferred embodiment, instead of random selection, all current f cluster centers are selected. avg The 25th, 50th, and 75th quantiles of the distribution of values ​​were used as three initial cluster centers, representing the initial anchor points for low-frequency, mid-frequency, and high-frequency data, respectively. A standard K-means iterative process was performed, continuously assigning each data point to the nearest center and recalculating the center point of each cluster until the change in the center point's position was less than a preset small value (e.g., 1 Hz), at which point the clustering was considered convergent. After clustering, all data items were naturally divided into three clusters. At this point, the dynamic stratification threshold θ... high and θ low This can then be determined based on the boundaries between clusters. For example, θ low All f values ​​in the low-frequency cluster can be retrieved. avg The maximum value and all f in the mid-frequency cluster avg The average of the minimum values. In this way, the hierarchical threshold is no longer rigid, but can perceive changes in the types and frequency distribution of connected devices throughout the system. For example, when more high-frequency devices are connected to the system, θ high It will automatically increase accordingly.

[0033] A preliminary hierarchical division is performed by combining the frequency mean of data items in the frequency feature vector set with the dynamic stratification threshold, and the hierarchical adjustment is performed based on the frequency stability of the data items to determine the hierarchical identifier.

[0034] In this embodiment, after obtaining the dynamic stratification threshold, a stratification determination is performed for each data item. Specifically, this includes a two-stage refined determination process: the first stage involves preliminary stratification. The average frequency f of the data items is then used as the stratification threshold. avg With the calculated dynamic stratification threshold θ high and θ low The comparison yields its initial tier. The second stage, based on frequency stability S... f Dynamic adjustments are made. While some data streams may have low average frequencies, they may contain sudden high-frequency pulses. Dividing data solely by average frequency might result in insufficient buffer capacity. Therefore, after initial layering, the system checks its frequency stability S. f =1 -σ f / μ f If S f Lower values ​​(e.g., S) f <0.7), while its frequency fluctuation range is relatively large (e.g., the difference between the maximum frequency and the average frequency f). max -f avgIf the frequency exceeds 30% of the average frequency, the system will determine that the data stream has unstable burst characteristics and elevate it to a higher level (e.g., from the mid-frequency layer to the high-frequency layer). This hierarchical decision-making considers not only the average case but also the worst-case scenario, enhancing the system's robustness.

[0035] Furthermore, to avoid frequent level switching (i.e., jittering) of data items near level boundaries due to minute frequency fluctuations, a boundary fuzzing mechanism is introduced. Specifically, for its f avg Falling within a narrow interval near the threshold (e.g., θ) high For data items within ±10Hz, the system calculates the normalized distance ratio r = d1 / d2 to its two adjacent cluster centers. If this ratio r is close to 1 (e.g., within the range of 0.8 < r < 1.2), the data item is marked as boundary data. For marked boundary data, the system employs a hysteresis strategy, meaning that a hierarchy switch is only performed when multiple consecutive decision cycles (e.g., 3 times) indicate that it should belong to a new hierarchy. This effectively suppresses jitter and ensures the stability of data routing.

[0036] Based on the hierarchical identifier, the data in the original data stream is allocated to different queues to form hierarchical data queues.

[0037] For example, after determining the final level identifier of each data item, it needs to be guided to a memory buffer queue tailored to it. Specifically, in this embodiment, different queue data structures are configured for different levels of data to optimize resource utilization and processing efficiency: A ring buffer queue is configured for high-frequency data based on the level identifier. High-frequency data streams are characterized by dense data points and a constant rate. The ring buffer queue is a fixed-length, efficient circular queue whose read / write pointers can move without locks or with very low contention, making it very suitable for this producer-consumer model. It can minimize the CPU overhead in data enqueue operations and ensure the highest data reception throughput. The queue capacity can be adaptively set according to the number of high-frequency data items, for example, Q. high = 10000·n high , where n high Q represents the number of high-frequency data items. high The capacity (i.e., queue length or maximum number of data items that can be accommodated) of the circular buffer queue configured for high-frequency level data items.

[0038] Configure a double buffer for intermediate frequency (IF) data: IF data exhibits both high data density and processing complexity. The double buffer mechanism comprises two buffers: a write buffer and a processing buffer. When the write buffer is full, the system immediately switches to the other empty buffer, writing new data to it. Simultaneously, the previously full buffer can be easily reordered by a background thread, with neither operation interfering with the other. This ping-pong mechanism ensures continuous data reception while providing lock-free, complete data blocks for time-consuming processing operations.

[0039] Configure a priority queue sorted by timestamp for low-frequency data: Low-frequency data is characterized by its sparseness and sporadic occurrence. Using a priority queue, with the timestamp of the data point as the priority, ensures that the queue remains chronologically ordered each time new data is added. Due to the sparse data volume, the overhead of insertion operations is entirely acceptable, while the benefit is that the low-frequency data queue is naturally pre-ordered, simplifying subsequent processing.

[0040] Alternatively, in another implementation, the capacity of all queues is not fixed but is monitored in real time. When the system detects that the occupancy rate of any queue continuously exceeds a safety threshold (e.g., 80%), it can trigger a dynamic expansion mechanism for the queue, or send a backpressure signal to the upstream data access layer, requesting it to appropriately reduce the data transmission rate, thereby forming a closed-loop, adaptive buffer management system.

[0041] This embodiment achieves this by introducing an adaptive hierarchical mechanism based on the inherent time scale of the data (i.e., sampling frequency). Specifically, the system extracts the frequency mean f for each data stream using a sliding window. avg and frequency stability S f The feature vectors quantify the invisible sampling speed and sampling regularity characteristics of the data. These vectors are then analyzed using the K-means clustering algorithm to dynamically divide the data into high, medium, and low frequency levels, rather than relying on fixed, empirical thresholds. This process allows the hierarchical division to adapt in real-time to changes in the overall frequency distribution of the data sources within the system. After initial stratification, the system utilizes frequency stability S... fFor bursty data streams with low average frequencies but transient pulses, hierarchical elevation is implemented. Based on the determined hierarchy, an optimized buffer queue structure is configured for the data stream (e.g., a circular buffer queue is used for high-frequency layers). This resolves the performance conflicts caused by a one-size-fits-all approach to storage due to neglecting sampling frequency differences. In smart energy scenarios, this means the system can automatically identify and isolate high-frequency data streams such as power quality monitoring, directing them to dedicated low-overhead processing channels. This prevents their intensive I / O operations from interfering with the stable storage of low-frequency data streams such as light intensity, thus achieving on-demand allocation of processing resources and global performance optimization at the initial stage of data access, improving the overall data throughput and stability of the system.

[0042] According to one aspect of this application, in a typical smart energy monitoring system, the local clocks of devices from different manufacturers and eras may deviate, and the network paths through which data is uploaded may also differ, resulting in varying timestamps for data generated at the same time. To eliminate this timing noise, such as... Figure 3 As shown, generating a virtual alignment index includes:

[0043] A discrete virtual time axis is created based on the minimum sampling interval of the system.

[0044] In this embodiment, the virtual timeline, also known as a virtual time grid network, is a unified, discrete, and regular time scale set for all accessed data. Specifically, it is necessary to determine the starting point t0 and the minimum scale Δt of this scale. min The starting point t0 can be flexibly set; for example, it can be set to the moment the system first starts, or a specific aligned moment (such as midnight of the day) for ease of management. The minimum scale, i.e., the system's minimum sampling interval Δt... min This is not a globally fixed static value, but a dynamically calculated parameter to adapt to changes in the data source within the system. In a preferred embodiment, the system periodically scans the frequency feature vector of all current data items to find the maximum sampling frequency f. max_all Theoretically, the minimum time interval should be 1 / f. max_all However, to balance time precision and metadata storage overhead, the system sets an engineering lower limit, such as 1 millisecond. Therefore, Δt min The calculation formula is Δt min = max(1 / f max_all (1ms). In this way, the highest resolution of the virtual timeline can always match the finest data source in the current system, while avoiding the problem of the entire timeline granularity becoming too fine due to occasional microsecond-level data.

[0045] In an alternative embodiment, such as Figure 4As shown, a discrete virtual timeline is created, specifically as follows:

[0046] A multi-layer virtual lattice network with increasing time granularity is constructed. This multi-layer virtual lattice network includes at least three frequency layers: a high-frequency layer, a mid-frequency layer, and a low-frequency layer. Specifically, for high, mid, and low frequency layers, a multi-layer virtual lattice network with increasing time granularity is constructed. The high-frequency layer uses the minimum sampling interval as the reference granularity, the mid-frequency layer uses N times the reference granularity, and the low-frequency layer uses M times the reference granularity, where M > N > 1. Preferably, N is 10, and M is 100.

[0047] In this embodiment, if all frequency data are forcibly mapped to a value of Δt... min On the highest precision time axis as the baseline, most time cells for low-frequency data will be empty, resulting in a significant waste of index space. To address this, a multi-resolution virtual grid network was constructed. Specifically, based on the baseline time granularity Δt... min The system will generate three independent virtual time grid networks: a high-frequency virtual grid network VT. high Its time granularity is Δt high = Δt min Intermediate Frequency Virtual Grid Network (VT) mid Its time granularity is Δt mid = 10·Δt min Low-frequency virtual lattice network (VT) low Its time granularity is Δt low = 100·Δt min Accordingly, each data item is mapped only to the virtual grid network corresponding to its own level. This is analogous to equipping maps of different scales with appropriately spaced latitude and longitude grids, ensuring the accuracy of high-frequency data while also considering the storage economy of low-frequency data. It should be noted that 10 and 100 here are exemplary multiples; in other implementations, they can be set to other reasonable, incremental values ​​based on the actual distribution of data frequency clusters, such as powers of 2 which are more computer-friendly.

[0048] By using a mapping function, the actual timestamps of each data point in the original data stream are converted into virtual grid positions on a virtual timeline, generating a virtual alignment index.

[0049] In this embodiment, after establishing the virtual timeline, it is necessary to process each arriving data with an actual timestamp t. realThe data points are precisely placed into their corresponding virtual cells. The mapping function is not a simple rounding operation, but introduces a dynamic offset compensation for each data item. Specifically, converting the actual timestamp to a virtual cell position includes: calculating the offset compensation based on the frequency stability and average delay of each data item in the frequency feature vector set; integrating the offset compensation into the mapping function to correct the actual timestamp; and then converting the corrected actual timestamp to a virtual cell position. The specific form of this adaptive mapping function is: g i (t real )=└(t real -t0+Δ i ) / Δt layer ┘;where g i (t real ) represents the virtual cell position calculated for a data point of data item i; t real t0 is the original timestamp of this data point; t0 is the starting point of the time axis; Δt layer The virtual cell time granularity of the level to which this data item belongs. Offset compensation amount Δ i The calculation formula is Δ i =d i *·(1-S f ); where d i * represents the historical average delay of data item i, expressed as d. i *= mean(t arrive - t data The calculation yields t, where t arrive t represents the timestamp of a data point actually arriving at the system (or receiving node). data This is the timestamp at which the data point was sampled or generated on its original acquisition device. i The * item directly compensates for the inherent, stable latency of the data stream (such as internal device processing latency, specific network path latency). And S f This represents the frequency stability of the data item. (1-S) f It plays a regulating role: if the clock of a data source is very stable (S... f The fact that the sampling time is close to 1 indicates that the sampling time is very regular, so the compensation amount Δ i For smaller timestamps, the original timestamp is more reliable during mapping. Conversely, if a data source experiences significant clock jitter (S... f (lower), then Δ i This will increase accordingly, providing a larger correction amount. This is achieved through dynamic compensation Δ i The mapping function effectively solves problems such as clock asynchrony between different devices and network latency jitter, and realizes the conversion from nominal time to real logical time, providing a solid foundation for subsequent high-quality data fusion and analysis.

[0050] This embodiment achieves this by constructing a unified, multi-resolution virtual time reference system and designing a mapping function with adaptive compensation. Specifically, the system constructs a high-precision virtual time axis based on the highest global sampling rate, and derives sparse time axes with progressively increasing granularity for medium and low-frequency data, forming a multi-resolution network. This ensures alignment accuracy while saving metadata storage space. The adaptive offset compensation amount in the timestamp mapping function is jointly determined by the historical average network delay and frequency stability of the data stream. The historical average network delay compensates for the inherent transmission delay, while frequency stability adjusts the strength of the compensation, providing stronger correction for devices with unstable clocks. In specific energy management applications, this means that the system can accurately align multiple phasor measurement unit (PMU) data from different substations with microsecond-level clock deviations to the same logical time point the instant the data is entered into the database. This allows upper-layer applications to directly perform high-precision wide-area power grid situational awareness and fault diagnosis analysis without performing complex, time-consuming, and error-prone secondary alignment, improving data availability and the reliability of analysis results.

[0051] According to one aspect of this application, in order to accommodate potentially late data, the system needs a time window for buffering. If the window is too small, late data will be prematurely judged as out of order; if the window is too large, it will increase memory usage and data processing latency. Therefore, an elastic window that can automatically expand and contract according to the punctuality of data arrival is adopted. Specifically, elastic buffering and out-of-order reordering are performed, including determining the size of the elastic window, as follows:

[0052] The arrival interval of data in the hierarchical data queue is monitored in real time, and the mean and standard deviation of the arrival interval are calculated. Based on the ratio of the standard deviation to the mean, and in combination with the preset elasticity coefficient, the baseline window is dynamically adjusted to determine the size of the elasticity window.

[0053] In this embodiment, the system continuously monitors the arrival time difference Δt between adjacent data points in each data queue. i =t i - t i-1 The smoothed mean Δt* and standard deviation σ(Δt) are calculated using methods such as exponentially weighted moving average (EWMA). In one possible implementation, the basic elastic window size W(t) can be given by the formula W(t) = W base The expression ·(1+α·σ(Δt) / Δt*) is given. Where W base The baseline window size is α (e.g., 500ms), α is the elasticity coefficient, and the ratio σ(Δt) / Δt* (i.e., the coefficient of variation CV) intuitively reflects the regularity of data arrival. The more irregular the data, the larger the ratio and the larger the window.

[0054] In another possible implementation, the elasticity coefficient is not a preset value, but is dynamically determined through the following steps: calculating the coefficient of variation of the arrival interval, which is obtained by dividing the standard deviation by the mean; and nonlinearly adjusting the basic elasticity coefficient based on the difference between the coefficient of variation and the target coefficient of variation to generate a dynamically determined elasticity coefficient.

[0055] In a more refined implementation, the elasticity coefficient α is itself adaptive. Its calculation formula is α(t) = α base ·(1+β·tanh(CV - CV target )); where α base Based on the basic elasticity coefficient (e.g., 0.3), CV target Let α(t) be the desired system stability target (e.g., 0.5), and β be the adjustment factor (e.g., 0.5). tanh is a sigmoid function; when the actual coefficient of variation (CV) is much larger than the target, α(t) will increase smoothly, making the system's response to fluctuations more aggressive; conversely, it will be more conservative.

[0056] In an optional embodiment, the dynamically adjusted elastic window size is smoothed by: fusing the currently calculated elastic window size with the elastic window size at the previous moment to generate a smooth window size, suppressing drastic changes in window size; applying hard maximum and minimum boundaries to the smooth window size, and applying soft constraints when the smooth window size approaches the maximum or minimum boundaries, to produce an elastic window size for buffer control.

[0057] In this embodiment, a smoothing function W is used to prevent sudden and drastic changes in window size. smooth (t) = γW(t) + (1-γ)W(t-1) (e.g., γ=0.7). Where W is the smoothing window size. smooth (t) Apply a hard boundary W min and W max (For example, 0.5 W) base and 3·W base This prevents memory usage from expanding indefinitely and ensures basic buffering capabilities. When approaching the boundary, a soft limit can be applied using the sigmoid function, making the window size change more gradual as it approaches its extreme value.

[0058] This embodiment employs a refined elastic window calculation method incorporating dual feedback adjustment, smoothing, and safety boundaries. Specifically, the buffer window size is dynamically adjusted based on the coefficient of variation (CV) of the data arrival interval, achieving the first layer of feedback. A second layer of feedback is introduced: the elasticity coefficient α(t) itself is also dynamic, adjusting according to the current CV and the system target CV. targetThe system performs non-linear adjustments to the deviation. This means that the system can not only respond to current jitter but also adjust its sensitivity to future jitter. The calculated window size is also smoothed and subject to strict upper and lower boundary constraints. This allows the buffer's memory usage and waiting latency to reach a dynamic optimal balance between dealing with network jitter and ensuring data order. In complex network environments such as distributed photovoltaic monitoring, some inverters may experience intermittent data transmission due to signal interference. This embodiment can automatically expand the buffer window for these problematic data streams, allowing longer waiting times to reassemble out-of-order data, while maintaining a minimum window for other stable data streams to reduce latency. This improves the system's robustness in data processing under unstable network environments and its overall real-time performance.

[0059] Optionally, elastic buffering and out-of-order reordering are performed, including dynamically reserving storage locations, specifically:

[0060] We study the delay distribution pattern of historical data in a hierarchical data queue, construct a delay distribution model and extract delay distribution parameters by analyzing the difference between data arrival time and data timestamp.

[0061] Specifically, the system will continuously learn the delay sequence d for each data item. i = t arrive - t data And the mean μ of its delay is calculated using a sliding window. d σ d 2 Furthermore, by constructing a delay distribution model and extracting the delay distribution parameters, the skewness γ of the delay distribution will also be extracted. d Skewness γ d The description of the symmetry of the delay distribution is crucial for understanding whether the data tends to arrive early or whether a small number of data points arrive extremely late.

[0062] Based on the delay distribution parameters, combined with the pre-stored current write position and data sampling frequency, the dynamic reserved position of future data points is calculated to guide the pre-layout of data in the buffer.

[0063] Furthermore, the specific steps for calculating the dynamically reserved position are as follows: Based on the skewness of the delay distribution, a dynamic adjustment coefficient is determined; the standard deviation of the delay distribution parameters is weighted using the dynamic adjustment coefficient, and combined with the mean of the parameters, the dynamically reserved position is calculated. In an exemplary embodiment, the formula for calculating the reserved position is: P expected =P current +┌(μ d + kσ d )·f sample ┐; where P current f is the write point in the current buffer.sample σ represents the sampling frequency of this data item, ┌ ┐ represents the rounding up sign, and σ represents the sampling frequency of this data item. d μ is the standard deviation of the delay. d This is the mean of the delay. The key lies in the dynamic adjustment coefficient k, which is determined by the skewness γ. d Decision: k = min(2, max(0.5, γ) d When the delay distribution is right-skewed (γ) d When the value of k is greater than 0 (meaning there is a small amount of extremely delayed data), the value of k will increase, making the reserved position further back, leaving more space for these burdensome data, thereby reducing the probability that they will be misjudged as overflow data.

[0064] Optionally, elastic buffering and out-of-order reordering are performed, further including adaptive hole detection, specifically:

[0065] Based on the standard deviation and data sampling frequency in the delay distribution parameters, an adaptive hole threshold is calculated and generated; the sequence consisting of dynamically reserved positions is scanned, and regions in the sequence where the interval between adjacent positions exceeds the adaptive hole threshold are identified as holes and managed.

[0066] In this embodiment, some unused gaps may arise in the reserved position sequence due to dynamic changes in the value of k. The system will determine the gaps based on formula θ. adaptive = max(3,┌3σ d ·f sample ┐) Calculate an adaptive hole threshold. When the size of consecutive holes exceeds this threshold, the system records them in the hole mapping table for subsequent memory cleanup or to fill other data.

[0067] Traditional location prediction often relies on the mean and variance of delay, while this embodiment introduces the skewness of the delay distribution as a decision factor. Skewness specifically corresponds to the scenario in the energy Internet of Things where most data transmission is normal, but a small number of data experience extreme delays due to routing anomalies or device restarts. A dynamic adjustment coefficient based on skewness is used to weight the standard deviation, thereby calculating the reserved locations. When the system detects a significant right skew in the delay (the presence of extremely late-arriving data), the adjustment coefficient automatically increases, reserving more redundancy space than in the case of a symmetrical distribution. Before the disordered reordering, most data (including those with predictable extreme delays) has already been placed near its final sorted position. This reduces the data movement distance and the number of comparisons in subsequent merge sort operations, lowers CPU computational overhead, and improves system processing efficiency.

[0068] Optionally, elastic buffering and out-of-order reordering are performed, further including intelligent overflow determination for handling out-of-order data, specifically:

[0069] When the deviation between the actual location of the data and the reserved location exceeds the threshold, the overflow priority is calculated by comprehensively considering the magnitude of the location deviation and the waiting time of the data in the buffer. Based on the overflow priority, the data is allocated to the main overflow buffer or the secondary overflow buffer to achieve hierarchical overflow management.

[0070] Specifically, when the actual arrival location of a piece of data deviates significantly from its reserved location, it requires special handling. This embodiment does not employ a simple binary decision-making process, but instead calculates the overflow priority: priority = |deviation| / σ d + age / T window The priority takes into account the relative magnitude of the deviation (|deviation| / σ). d The data's waiting time in the buffer is T, where |deviation| is the absolute deviation between the actual arrival position of the data and its reserved position. window A time window is set for the system. High-priority data (i.e., data with large deviations or long waiting times) is placed in the main overflow buffer and has the right to be reassembled first; lower-priority data is placed in the secondary overflow buffer. This hierarchical management makes the reassembly process more intelligent.

[0071] Optionally, elastic buffering and out-of-order reordering are initiated by a multi-condition triggering decision, which includes:

[0072] The monitoring buffer is triggered by conditions, and out-of-order reassembly is initiated when any one of the conditions of time triggering, capacity triggering, or out-of-order degree triggering is met; the condition of out-of-order degree triggering is that the proportion of overflow data exceeds a preset out-of-order threshold.

[0073] For example, when the out-of-order reordering operation (e.g., merge sort) of the buffer is initiated is determined by several conditions: Time-triggered: The current time minus the window start time exceeds the final calculated elastic window size W. final Capacity trigger: The total amount of data in the buffer (main buffer + overflow area) exceeds a preset threshold, such as 1MB. Out-of-order trigger: The proportion of data in the overflow buffer exceeds the out-of-order threshold, such as 30%, indicating that the current data stream's disorder has severely impacted buffer efficiency and requires immediate reorganization. Reorganization is initiated when either condition is met. Additionally, as an optional implementation, the system also monitors the backlog duration of the overflow area. If the backlog exceeds a longer hard time limit (e.g., 3 times the window size), forced merging is triggered, and a rate-limiting signal can be sent upstream to prevent the system from being overwhelmed by continuously severe out-of-order data.

[0074] In an optional embodiment, prior to the parallel persistence of ordered data blocks, differential storage encoding is further included, specifically: differential storage encoding is performed separately for ordered data blocks belonging to different frequency layers of high, medium and low frequencies.

[0075] In this embodiment, the study found that data of different frequencies have vastly different numerical distributions and variation characteristics, and a one-size-fits-all compression or storage method cannot achieve optimal results. Specifically, the encoding strategy is as follows: Delta encoding is applied to high-frequency layer data blocks: High-frequency data (such as power grid phase angles) are usually continuously changing, and the numerical differences between adjacent sampling points are very small. Delta encoding takes advantage of this characteristic, not directly storing the absolute value of each data point, but storing the absolute value of the first point and the difference (i.e., increment) between all subsequent points and the previous point. For example, a string of original data [100.1, 100.2, 100.3, 100.1] can be stored as [100.1, +0.1, +0.1, -0.2] after Delta encoding. Since the increment value is usually very small, it can be represented with fewer bits, thereby achieving a very high compression ratio.

[0076] Extended space is reserved for intermediate frequency (IF) layer data blocks: IF data (such as device power) has a certain degree of stability, but its records may be updated or have additional information (such as quality markers, calibration instructions, etc.) added in the future. If compact storage is used, any minor modification may lead to the reallocation and migration of the entire data block, triggering write amplification. Therefore, in this embodiment, when persisting IF data blocks, in addition to storing their original format, an additional portion (e.g., 20%) of extended space is reserved. This trade-off of space for time improves the efficiency of subsequent data updates.

[0077] Low-frequency data blocks are stored using a sparse timestamp-value pair format: Low-frequency data (such as daily cumulative electricity consumption) is characterized by a large time span but sparse data points. Allocating contiguous storage space for it would be extremely wasteful. Therefore, this embodiment only stores valid (timestamp, value) pairs. This sparse format is the most compact and efficient storage method for this type of data.

[0078] Furthermore, after encoding, the ordered data blocks are persisted in parallel. In this embodiment, to avoid mutual interference between data write operations at different levels, especially to prevent high-frequency data streams from starving low-frequency data streams due to intensive I / O, a scheduling strategy based on logical partitions and independent I / O channels is adopted. For example, in a server with multiple SSDs, the logical partition carrying high-frequency data can be bound to an independent SSD, while the partitions carrying medium and low-frequency data share another SSD. The I / O operation adopts an asynchronous write mode, so that the CPU does not need to wait for the write instruction to complete after issuing it, and can continue to process other tasks, thereby maximizing the system throughput.

[0079] Furthermore, to enable rapid information retrieval from massive amounts of data, an efficient index is constructed. Optionally, a multi-dimensional index tree containing frequency, time, and location dimensions is built, preferably implemented as a B+ tree with triples as keys, specifically:

[0080] The physical address, timestamp, and frequency level associated with the data are mapped to discretized location coordinates, time coordinates, and frequency coordinates, respectively, to generate a three-dimensional coordinate set.

[0081] Specifically, for each persisted data point, the system will extract or generate a three-dimensional coordinate (t). coord f coord p coord Time coordinate t coord : Through formula t coord =└ (timestamp - t base ) / Δt index ┘Calculation; where timestamp is the actual timestamp of the data point, t base It is a global time base, t index This refers to the time granularity of the index, which can be set according to query requirements. For example, setting it to 1 second means that all data within the same second will have the same time coordinate in the index. Frequency coordinate f coord The most direct way is to use the layer identifier it belongs to. id (For example, 0 represents low frequency, 1 represents mid frequency, and 2 represents high frequency). In another, more refined implementation, to support more complex frequency range-based queries, a logarithmic mapping can be used: f coord = └ log2(frequency / f base ) ┘, where frequency is the average frequency of the data item, f base This is the fundamental frequency (e.g., 1Hz). Logarithmic mapping can effectively handle a wide frequency range from <1Hz to >1kHz. Position coordinates p coord : It is calculated from the starting address of the data block on the physical storage medium, physical_addr, for example, through a right shift operation p coord = physical_addr >> 12, align it with the page size of the storage system (e.g., 4KB), and reduce the cardinality of the index, where >> means shift the address to the right.

[0082] An index tree is constructed based on a three-dimensional coordinate set. When the index node is split, the variance of the data distribution of the coordinates within the node in each dimension is analyzed, and the dimension with the largest variance is selected as the split dimension to generate a multi-dimensional index tree.

[0083] In this implementation, the system uses a B+ tree as the basic index structure, and its key values ​​are composite keys composed of the aforementioned three-dimensional coordinates. To optimize query performance, the dimensional order of the composite key is crucial. Preferably, the key order is set as (t... coord f coord p coord The time dimension is prioritized because most time-series data queries involve time range filtering. This key order allows the B+ tree to quickly locate the starting point of the time range and then perform continuous scanning. Traditional B+ trees typically split in the middle when a node is full. However, this embodiment calculates the variance of the data distribution across the three dimensions of time, frequency, and location for all three-dimensional coordinate points within that node when splitting is necessary; this variance is σ. t 2 , σ f 2 , σ p 2 The dimension with the largest variance is selected as the dimension for this split, and the median of that dimension is selected as the split point. For example, if the data within a node has a large time span (σ... t 2 If the data is at its maximum (but all belong to the same frequency layer), the system will choose an intermediate time point for splitting. This allows the index structure to self-optimize based on the distribution characteristics of local data, effectively combining the range scanning advantages of B+ trees with the multi-dimensional partitioning idea of ​​kd trees, thus providing stable and efficient performance for various complex joint queries.

[0084] In this embodiment, the internal structure of the index is no longer preset and static, but can self-optimize based on the actual distribution of the data it carries. The resulting index tree exhibits robust high performance for various combined queries. Whether it's a regular query based on a time range or a special query filtering specific frequency layers, this index provides a near-optimal query path, avoiding the huge overhead of building and maintaining multiple single indexes for different query modes, thus improving the query response speed of the energy data analysis platform and the overall maintainability of the system.

[0085] In summary, this application discloses an intelligent data storage method for smart energy units, aiming to solve the problems of low data storage efficiency and low query performance for massive heterogeneous, multi-frequency time-series data. The method includes: analyzing the sampling characteristics of the original data stream to generate a frequency feature vector set for each data stream, containing the frequency mean, maximum value, and stability; based on the frequency feature vector set, using a clustering algorithm for adaptive hierarchical routing to divide the data stream into different levels such as high, medium, and low frequencies, and constructing a multi-granularity virtual time grid network to map the data onto a unified time axis to achieve time-series alignment; for each layer of data queues, using an elastic time window based on the statistical characteristics of data arrival intervals for efficient out-of-order data buffering and reorganization; and persistently storing the ordered data blocks of each layer in parallel using differentiated encoding strategies (such as Delta encoding, sparse format, etc.), and integrating metadata to construct a multi-dimensional B+ tree index containing frequency, time, and location dimensions. This invention can optimize the configuration of storage resources and adaptively adjust the processing flow according to the inherent sampling frequency characteristics of the data, improving the storage density and query analysis efficiency of massive heterogeneous energy data.

[0086] In optional embodiments, under certain extreme conditions, such as when the system has just started, when the data sample is extremely small, or when the data frequency distribution is ill-conditioned (e.g., all data frequencies are squeezed into a very small range), the K-means algorithm may fail to converge or produce meaningless clustering results (e.g., category collapse). To address this situation, the system of this application incorporates a degradation path. The system continuously monitors the health of K-means clustering. For example, if it finds that the number of members in a cluster is zero, or that the cluster center experiences severe oscillations for multiple consecutive iterations, the system will determine that the dynamic stratification has failed. In this case, the system will not interrupt the service but will automatically switch to a safe mode. In this mode, the stratification decision will degenerate to using a set of pre-configured, industry-experience-based fixed frequency thresholds (e.g., <10Hz for low frequency, 10-100Hz for mid frequency, and >100Hz for high frequency). Simultaneously, the system will continue to attempt to execute the K-means algorithm in the background. Once the subsequent data distribution returns to normal, allowing the algorithm to produce stable results, the system will automatically switch back to the superior dynamic stratification mode.

[0087] In an optional embodiment, if a data source continuously sends severely out-of-order data, the overflow area may remain in a backlog state for an extended period, continuously consuming significant amounts of memory and CPU resources for reorganization. To address this, the system employs a combined mechanism of forced merging and upstream rate limiting. A separate monitoring thread periodically checks the status of the overflow area. If the occupancy rate of an overflow area is found to be consistently above the danger level (e.g., 90%) for a prolonged period (e.g., three times the current elastic window size), the system triggers a forced merging operation to immediately clean up the backlogged data and release memory. Furthermore, simultaneously with triggering the forced merging, the system sends a backpressure or rate limiting signal to the upstream of the data stream (data acquisition layer or gateway). This signal requests the upstream to temporarily reduce the acquisition frequency of the data item or to implement additional buffering at the gateway level. This upgrade from passive processing to proactive intervention effectively prevents the risk of a single problematic data source dragging down the entire storage system.

[0088] In an optional embodiment, Write-Ahead Logging (WAL) technology is preferably used to ensure the atomicity and durability of data writes. The specific process is as follows: When an ordered data block that has been reorganized from a disordered state needs to be persisted, the system first writes a log record describing the operation (including data content, location mapping relationships, index changes, etc.) to a separate, sequentially read-write WAL file, and then safely flushes this log record to disk. Only then does the system actually modify the physical data block, location mapping table, and multidimensional index tree. If the system crashes (e.g., due to a power outage) during physical modification, upon system restart, it first checks the WAL file. The recovery process reads the log and finds transactions that have been recorded but not marked as completed. For these incomplete transactions, the system re-executes the corresponding physical write operation (i.e., redo) based on the log content. Since all write operations are designed to be idempotent (i.e., repeating them multiple times has the same effect as executing them once), even if some operations were partially completed before the crash, the redo operation can safely restore the system to a completely consistent state after the transactions were completed. This ensures that there will never be an intermediate state where the data and the index are inconsistent, thus guaranteeing the integrity and reliability of the data.

[0089] In a specific embodiment, assuming the application is to a mid-frequency data stream processing scenario, for example, a data item is the output voltage of an inverter, whose theoretical sampling frequency f sample The frequency is 50Hz (theoretically one data point every 20ms). The system has configured the following initial parameters for the intermediate frequency layer where this data stream resides: reference window size W. base = 500 ms; Hard bounds for window size: W min = 250 ms, W max= 1500 ms; basic elastic coefficient α base = 0.3; Elasticity coefficient adjustment factor β = 0.5; Target coefficient of variation CV target = 0.5; Window smoothing coefficient γ = 0.7; Final window size W(t-1) of the previous time step = 510 ms; Logical write position P in the current buffer. current = 1000. The system collected two sets of key time-series feature sequences from the most recent 10 data points: delayed time series d i (Unit: ms), i.e., t arrive - t data The sequence [15, 12, 18, 14, 16, 25, 13, 17, 19, 80] shows a significant delay of 80 ms. The arrival time interval sequence Δt... i (Unit: ms), i.e., t i - t i-1 [20, 22, 18, 19, 25, 15, 23, 17, 28, 21]. Based on delayed time series d i Calculate the reserved position in the buffer for the next data point to arrive. Based on the given delay sequence d... i = [15, 12, 18, 14, 16, 25, 13, 17, 19, 80], calculate its statistical parameters: mean delay μ d μ d = (15+12+18+14+16+25+13+17+19+80) / 10 = 22.9 ms. Delay standard deviation σ d σ d =sqrt(1 / 10∑ i=1 10 (d i -μ d ) 2 )≈20.2ms. Delay distribution skewness γ d γ d = (1 / 10∑ i=1 10 (d i -μ d ) 3 ) / σ d 3 ≈2.4. It can be seen that the skewness γ d A value much greater than 0 indicates a significant right skewness in the delay distribution, meaning that most data points have relatively small delays, but a few extreme cases of very large delays exist. This can be analyzed using the skewness γ. d To determine the dynamic adjustment coefficient k, the calculation formula is k = min(2, max(0.5, γ). d Substitute γd = 2.4, resulting in k = min(2, max(0.5, 2.4)) = min(2, 2.4) = 2.0. This calculation shows that, due to the system's identification of the significant right skewness of the delay distribution, it chose the largest adjustment coefficient k = 2.0, implying a more conservative approach in reserving positions, providing ample buffer space for potentially extreme delay data. Calculate the dynamic reserved position for the next data point: P expected = P current +┌(μ d + kσ d )·f sample Substitute the value: P expected =1000 +┌(22.9ms+2.0·20.2ms)·(50 points / 1000 ms)┐=1000 + 4 = 1004. The final calculation shows that the system will reserve space for the next data point at the 4th position after the current write point (i.e., position 1004).

[0090] In parallel, the system is based on the arrival time interval sequence Δt i Calculate the appropriate elastic window size for the current time step. This is based on the given arrival interval sequence Δt. i = [20, 22, 18, 19, 25, 15, 23, 17, 28, 21], calculate its statistical parameters: Interval mean Δt*: Δt* = (20 + 22 + 18 + 19 + 25 + 15 + 23 + 17 + 28 + 21) / 10 = 20.8 ms. Interval standard deviation σ(Δt): σ(Δt) = sqrt{1 / 10∑ i=1 10 (Δt i -Δt*) 2 The time to 3.9 ms is approximately equal to 3.9 ms. The coefficient of variation (CV) is calculated as: CV = σ(Δt) / Δt* = 3.9 / 20.8 ≈ 0.1875. This is based on the CV and the target CV. target The difference is used to nonlinearly adjust the basic elastic coefficient. α(t) = α base ·(1 + β· tanh(CV - CV targetSubstituting the values: α(t) = 0.3·(1 + 0.5·tanh(0.1875 - 0.5)) = 0.3·(1 + 0.5·tanh(-0.3125)). Consulting the hyperbolic tangent function table or calculating, we know that tanh(-0.3125)≈-0.302. Therefore, α(t) ≈0.3·(1 + 0.5·(-0.302)) = 0.3·(1 - 0.151) = 0.3·0.849≈0.255. This result indicates that the arrival regularity of the current data stream (CV=0.1875) is better than the system's target (CV). target =0.5), the system adaptively lowers the elasticity coefficient, making the window size more responsive to fluctuations. Calculate the initial elastic window size W(t): W(t) = W base ·(1 + α(t)· CV). W(t) = 500·(1 + 0.255·0.1875) ≈ 500·(1 + 0.0478)≈523.9 ms. Smoothing: W smooth W(t) = γW(t) + (1-γ)W(t-1) = 0.7·523.9 + 0.3·510 = 366.73 + 153 = 519.73 ms. Boundary check: Check W smooth (t) Whether it is in [W min W max That is, within the interval [250, 1500]. 519.73ms falls within this interval and does not require a hard limit. Therefore, the final elastic window size W used for buffer control in this cycle is... final It was determined to be approximately 520ms.

[0091] According to one aspect of this application, in real-world industrial environments, data streams are often accompanied by anomalies such as noise, packet loss, or duplicate reporting. If the frequency feature extraction process is not robust enough, it will directly affect the accuracy of all subsequent layering and buffering strategies. In basic methods, the sliding window typically moves forward with a fixed step size. However, this unchanging strategy is inefficient or lacks smoothness when processing data streams with drastically changing density. In a preferred embodiment, the original data stream is processed using an adaptive window sliding strategy. Specifically, the sliding step size step is no longer a fixed value, but is dynamically calculated based on the data density within the window: step = max(S min T w ·(1 - density)); where S min This is the minimum step size, used to ensure basic analytical accuracy, such as 10 ms; T wThis refers to the window width, for example, 1000ms; density is the data density factor, calculated as density = N. current / N expected N current N represents the number of data points actually captured within the current window. expected This refers to the expected number of data points based on the frequency calculated from the previous period. In this way, when the data stream is very dense and stable (density close to 1), the sliding step size becomes very small, maintaining a high overlap between windows (e.g., 75% or higher), thus producing smooth and continuous frequency estimates. Conversely, when the data stream becomes sparse (lower density), the step size increases accordingly, allowing the analysis window to jump to the next data-dense area more quickly, avoiding ineffective calculations in large blank areas and improving processing efficiency.

[0092] According to one aspect of this application, in the basic method, the frequency is determined by the total number N of data points within the window. i Direct calculations show that this method is highly susceptible to outliers. For example, a network jitter might cause a batch of data to be reported repeatedly, resulting in N values ​​in a certain window being affected. i A sudden spike; or a brief device outage could cause N i Instantaneous zeroing can severely pollute the frequency calculation results. Therefore, in a preferred embodiment, a two-stage robust frequency estimation algorithm is employed: Stage 1: Outlier Removal. Within one analysis period, a sequence of data point counts [N1, N2, N3, ...] is obtained, calculated from multiple consecutive sliding windows. The mean N* and standard deviation σ of this sequence are then calculated. N ; Traverse the sequence and find all sequences that satisfy |N i - N*| >3σ N N i The value is considered a statistical outlier and is temporarily removed from the sequence. 3σ N Based on the 3-σ principle of normal distribution, it can effectively identify extreme anomalies. Phase Two: Estimation using the median. After removing outliers, for the remaining, purer quantity sequence, instead of the mean, the median (median(N)) is used for estimation. i The median serves as the final count estimate. It is insensitive to extreme values ​​at both ends of the sequence and better reflects the main trend of the data. The robust frequency within this period is calculated as: f robust = median(N i ) / T w As an alternative implementation, the outlier removal threshold does not have to be a fixed 3σ. NInstead, it adopts the IQR (Interquartile Range) criterion based on box plots, which removes values ​​that exceed the range of [Q1 - 1.5·IQR, Q3 + 1.5·IQR]. This is more robust for non-normally distributed data, where Q1 is the first quartile, representing the value at the 25th percentile of the data; Q3 is the third quartile, representing the value at the 75th percentile of the data; and IQR is the interquartile range, representing the middle 50% of the data distribution.

[0093] By combining adaptive sliding with robust estimation, this implementation can penetrate the noise fog on the data surface and obtain the true and stable sampling frequency characteristics of the data stream, providing a high-quality data foundation for subsequent decisions in the entire intelligent storage process.

[0094] In an alternative embodiment, to enable efficient use of the virtual alignment index, it is designed as a bidirectional hash index structure. This structure consists of two interrelated hash tables (or dictionaries): a forward index table (Index). grid Its key is the virtual cell position v. pos Its value is a list that stores the identifiers of all data items that fall within that time frame. id Its structure is as follows: v pos Mapped to [data] id1 , data id2 This index is used to answer queries such as "What data did the system collect at a given moment?". The inverted index table is called Index. reverse Its key is the data item identifier. id Its value is a list or bitmap that records which virtual cell positions contain the data item. Its structure is as follows: data id Mapped to [v pos1 v pos2 [...]. This index is used to answer the question, "At what times are the historical values ​​of a certain data item distributed?". By maintaining a bidirectional index structure, whether it is a time-based aggregation query or a backtracking query based on a specific data item, the system can quickly locate the required metadata with an average time complexity close to O(1), thus improving query efficiency.

[0095] In another alternative embodiment, in a multi-threaded, high-concurrency write scenario, the bidirectional index table, as a shared data structure, must have thread-safe update operations. The traditional approach is to use a global lock, but this makes index updates a performance bottleneck for the entire system. A preferred approach is to employ a copy-on-write (COW) concurrency control strategy. Specifically, when a worker thread needs to add a new record to the index (e.g., adding data...), the update operation must be thread-safe. id3 Add to v pos_A (in the list), it doesn't directly modify the original list. Instead: it creates a pointer to v. pos_A A copy of the original list; perform the addition of data on the newly created copy. id3 The operation; through a single atomic pointer swap operation, the Index is... grid Chinese v pos_A The key points to this new, modified copy of the list. During this process, threads reading the index always access the unmodified, consistent old list until the pointer swap is complete, at which point they access the new list. This mechanism avoids read-write conflicts and achieves lock-free concurrent reads. Furthermore, to reduce the overhead of frequent copy creation and pointer swapping, the system incorporates a batch update strategy. Each worker thread temporarily stores its index update operations in its local buffer. A background aggregation thread merges all thread update operations every fixed time interval (e.g., 100 milliseconds) and then updates the main index all at once using a copy-and-write (COW) strategy.

[0096] In an optional embodiment, the location mapping table M pos This records the mapping of data points from logical locations to physical storage locations. However, storing a (logical_pos, physical_pos) tuple for each data point (typically each location is a 64-bit integer) consumes a huge amount of storage space. To compress this metadata, this embodiment uses differential encoding for the location mapping table. Since both logical and physical locations are monotonically increasing when writing ordered data blocks, the difference between logical_pos and physical_pos of two adjacent data points is usually a small positive number (often 1). Differential encoding takes advantage of this; instead of storing the absolute value of each location, it stores the absolute position of the first data point and the difference between the position of each subsequent data point and the position of the previous data point, i.e., (ΔL). pos ΔP posThese often small differences can be stored using variable-length integer encoding (such as Varint), reducing the number of bits required for storage. For example, a 64-bit integer 1 might only require one byte when encoded using Varint. Through this embodiment, the size of the location mapping table can be compressed by more than an order of magnitude, reducing the system's metadata storage overhead.

[0097] In an optional embodiment, a high-performance query acceleration scheme is provided for the multidimensional index tree. Although the optimized B+ tree structure reduces disk I / O, any disk access can become a bottleneck in high-concurrency, low-latency query scenarios. This embodiment fully utilizes the temporal locality and access hotspot principle unique to time-series data queries. By constructing a sophisticated multi-level index cache in memory, the index traversal process for the vast majority of queries can avoid disk access, achieving a leap in query performance. Time-series data queries typically have the following characteristics: first, recent data is accessed much more frequently than historical data (temporal locality); second, data from a few key measurement points or star devices in the system is accessed much more frequently than data from other measurement points (access hotspots). The multi-level caching architecture of this embodiment is designed to match these characteristics. Specifically, this caching architecture consists of three levels (L1, L2, L3): L1 is a leaf node cache; this level is dedicated to storing the leaf nodes of the B+ tree. Leaf nodes are the endpoints of index traversal and directly contain addresses pointing to physical data blocks. The L1 cache is configured as a limited-capacity region that uses the Least Recently Used (LRU) algorithm for page replacement. For example, its capacity can be configured to the 32 most recently accessed leaf nodes. When a query needs to access a leaf node, the system first checks the L1 cache. If a match is found, the result is returned directly from memory; if a match is not found, the leaf node is loaded from disk and placed into the L1 cache (if the cache is full, the least recently accessed node is evicted according to the LRU policy). Due to temporal locality, most queries focus on the latest data, meaning that leaf nodes corresponding to the most recent time range will be accessed repeatedly. Keeping these hot leaf nodes in the L1 cache allows queries on recent data to require almost no disk I / O, reducing query latency to the microsecond level. L2 cache is an intermediate node cache; this level of cache stores non-leaf intermediate nodes of the B+ tree, which constitute the navigation path from the root to the leaves. Similar to L1, L2 cache is also managed using the LRU strategy, but its capacity is usually larger, for example, configured to cache 128 frequently accessed intermediate nodes. Not all queries only access the latest data; for queries that need to trace history or perform large-scale statistics, the access hotspot principle comes into play. For example, intermediate nodes corresponding to the start time of each month or day, and index paths corresponding to critical devices, are accessed with a very high probability. Keeping these frequently accessed intersection nodes in L2 cache can accelerate the index traversal speed of various historical queries. L3 cache is a top-level resident cache; this level of cache is used to permanently store the top few levels of nodes in the B+ tree, such as all nodes from the root node to a depth of 3. The contents of L3 cache are loaded when the system starts and are never replaced; the top-level nodes of the B+ tree are the necessary path for all queries, and every query must start from the root node.By storing the entry points for this part of the index in memory, it is ensured that the startup phase of any query is zero-latency, providing a fundamental guarantee for overall low-latency query performance.

[0098] As an optional optimization, this caching system can also perform cache warm-up operations during system startup or idle periods. For example, the system can analyze the most frequently accessed time ranges and data items based on historical query logs, and then proactively load the index nodes that these queries may involve into the L1 and L2 caches in advance, thereby avoiding performance fluctuations caused by cache cold starts when the system first starts providing services.

[0099] According to another aspect of this application, the intelligent data storage method for smart energy units can also be:

[0100] It receives raw data streams from heterogeneous energy terminals, analyzes the sampling characteristics of each data item through a sliding time window, and generates a frequency feature vector set and time-series metadata.

[0101] Dynamic hierarchical determination of data streams is performed based on frequency feature vector sets. At the same time, a virtual time grid network is constructed to map the original data streams of different frequencies to the corresponding storage layers, and output hierarchical data queues and virtual alignment indexes.

[0102] For the hierarchical data queues at each layer, an elastic time window mechanism is used to handle out-of-order data, and a location reservation strategy is used to optimize the storage layout to generate ordered data blocks and location mapping tables.

[0103] Ordered data blocks from each layer are written to physical storage in parallel according to a layer-specific storage strategy, while the three-dimensional index structure is updated to generate persistent data and a multi-dimensional index tree.

[0104] Receive query requests, locate data using multidimensional index trees and virtual alignment indexes, reconstruct data with different sampling rates using a virtual time grid alignment mechanism, and return the query result set.

[0105] In a preferred embodiment, a typical query request might be: query the change curves of inverter A's output power (high frequency), the ambient temperature of its location (low frequency), and the grid frequency (medium frequency) between 9:00 AM and 9:10 AM yesterday. The complete process for processing this query is as follows: The query engine first parses the request and extracts the time range [t]. start , t end ] and a list of data items [power, temperature, grid frequency]. Utilizing the constructed, in terms of (t coord f coord p coord ) is a B+ tree index for composite keys. The engine uses the first dimension t. coordA range scan is performed to efficiently locate all index entries whose timestamps fall within that range. Based on this, the second dimension f can be further utilized. coord The results undergo initial filtering. Thanks to the caching mechanism, the entire process is very fast. The final output is a list containing the addresses of all physical data blocks located at different storage levels that may contain the target data. Based on the obtained list of physical addresses, parallel asynchronous I / O requests are initiated to the storage layer to simultaneously read high, medium, and low-frequency data blocks from different physical partitions. Once the data blocks are read into memory, the system calls the corresponding decoder. For example, Delta decoding is performed on the high-frequency power data block to restore its original numerical sequence. For the low-frequency temperature data block, its (timestamp, value) pairs are directly parsed. Three original data sequences are output, each with its own original timestamp and a different sampling rate. To compare and analyze these three sets of data with vastly different frequencies, they must be unified onto the same time reference system. The system generates a unified high-resolution query timeline based on the sampling rate of the highest-frequency data in the query request (power data in this example). All three data sequences are then mapped onto this unified timeline. High-frequency data can basically fill most of the time grid, while medium- and low-frequency data will create a large number of regular or irregular gaps. In this case, the system analyzes the pattern of the gaps and selects the optimal interpolation algorithm: for regular gaps between two points caused by low sampling rates, linear interpolation, which has low computational cost and good performance, is used, i.e., v interp = v i + (v i+1 -v i )·(tt i ) / (t i+1 -t i ), where v interp v is the interpolated estimate at time point t. i For timestamp t i The original values ​​are used. For irregular gaps caused by data loss, a more complex cubic spline interpolation is used to ensure the smoothness and differentiability of the curve. For boundary gaps located at the query range boundary that cannot be bidirectionally interpolated, nearest neighbor filling is used, that is, the value of the nearest valid point is used to fill them. After interpolation, theoretically, each time point on the unified time axis may have one or more values ​​(from different sources or interpolations). Finally, fusion is performed to generate the final, clean result set. If there is only one value at a certain time point, it is used directly; if there are multiple values ​​(for example, a high-frequency original value and a low-frequency interpolation), a choice or fusion must be made. This embodiment uses a weighted average fusion strategy. Its fusion formula is v aligned =∑(w i ·v i ) / ∑w i, where v aligned The weight w is the unified value after merging. i The calculation method is as follows: w i =S f ·(1 / |t actual - t virtual |). S f This refers to the frequency stability of the data item from which the value originates. This gives higher weight to high-quality data sources with more stable and reliable clocks. actual - t virtual | represents the deviation between the original timestamp of the value and the time of its corresponding grid point. A true original measurement close to the grid point will have a much higher weight than an estimate obtained by interpolation between two distant points. Through weighted fusion, this embodiment ultimately outputs a high-fidelity multivariate time series that is time-aligned and numerically integrates the quality of the data source and its temporal proximity. This series can be directly used for precise visualization, correlation analysis, or as input to more advanced analytical models.

[0106] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. An intelligent data storage method for smart energy units, characterized in that, include: Analyze the sampling characteristics of the raw data stream from the smart energy unit to generate a set of frequency feature vectors; Adaptive hierarchical routing is performed based on the frequency feature vector set, and a virtual time grid is constructed to generate a hierarchical data queue and a virtual alignment index. Elastic buffering and out-of-order reordering are performed on the hierarchical data queue to produce ordered data blocks and a location mapping table; The ordered data blocks are persisted in parallel, and the location mapping table and virtual alignment index are merged to build a multi-dimensional index tree containing frequency, time and location dimensions; the parallel persistence means that ordered data blocks from different levels are written to the physical storage medium in parallel and asynchronously through independent I / O channels. Generate a hierarchical data queue, including: The K-means clustering algorithm is used to process the frequency feature vector set, and the dynamic stratification threshold is calculated. A preliminary hierarchical division is performed by combining the frequency mean of data items in the frequency feature vector set with the dynamic stratification threshold, and the hierarchical adjustment is performed based on the frequency stability of the data items to determine the hierarchical identifier. Based on the hierarchical identifier, the data in the original data stream is allocated to different queues to form hierarchical data queues; Generate a virtual alignment index, including: A discrete virtual time axis is created based on the minimum sampling interval of the system; By using a mapping function, the actual timestamps of each data point in the original data stream are converted into virtual grid positions on a virtual timeline, generating a virtual alignment index; Performing elastic buffering and out-of-order reordering includes determining the elastic window size, specifically: Real-time monitoring of the arrival interval of data in the hierarchical data queue, and calculation of the mean and standard deviation of the arrival interval; Based on the ratio of standard deviation to mean, and in conjunction with a preset elasticity coefficient, the benchmark window is dynamically adjusted to determine the size of the elasticity window; Perform elastic buffering and out-of-order reordering, including dynamically reserving storage locations, specifically: Learn the delay distribution pattern of historical data in a hierarchical data queue, construct a delay distribution model and extract delay distribution parameters by analyzing the difference between data arrival time and data timestamp; Based on the delay distribution parameters, combined with the pre-stored current write position and data sampling frequency, the dynamic reserved position of future data points is calculated.

2. The method according to claim 1, characterized in that, The data in the original data stream is distributed to different queues, specifically as follows: Based on the hierarchical identifier, a circular buffer queue is configured for high-frequency layer data, a double buffer queue is configured for mid-frequency layer data, and a priority queue sorted by timestamp is configured for low-frequency layer data.

3. The method according to claim 1, characterized in that, Create a discrete virtual timeline, specifically as follows: Construct a multi-layer virtual lattice network with incremental time granularity; the multi-layer virtual lattice network includes at least three frequency layers, namely a high-frequency layer, a mid-frequency layer, and a low-frequency layer; Among them, the high-frequency layer uses the minimum sampling interval as the reference granularity, the mid-frequency layer uses N times the reference granularity, and the low-frequency layer uses M times the reference granularity, where M>N>1.

4. The method according to claim 1, characterized in that, Convert the actual timestamp to a virtual grid location, including: The offset compensation amount is calculated based on the frequency stability and average delay of each data item in the frequency feature vector set. The offset compensation is incorporated into the mapping function, and after correcting the actual timestamp, it is converted into a virtual grid position.

5. The method according to claim 1, characterized in that, The determination of the elasticity coefficient includes: Calculate the coefficient of variation of the arrival interval, which is obtained by dividing the standard deviation by the mean; Based on the difference between the coefficient of variation and the target coefficient of variation, the basic elastic coefficient is nonlinearly adjusted to generate a dynamically determined elastic coefficient.

6. The method according to claim 1, characterized in that, Smoothing is applied to the dynamically adjusted elastic window size, specifically as follows: By combining the current elastic window size with the elastic window size from the previous time step, a smooth window size is generated. Apply hard maximum and minimum boundaries to the smooth window size, and apply soft constraints when it approaches the maximum or minimum boundaries, producing a flexible window size for buffered control.

Images