Data scheduling system and method for multi-layer storage architecture based on multi-core processor

Abstract

The application relates to the technical field of data processing, and discloses a multi-layer storage system data scheduling system and method based on a multi-core processor. The application improves the continuity and real-time response performance of a multi-core processor memory access link by acquiring a multi-core memory access event stream and identifying execution phase changes of processor cores. A data flow graph is constructed through address dependency and timing correlation analysis. Based on the topological characteristics of the data flow graph, inter-core access distribution, and effective access time length, the sharing degree, affinity, and survival period of a data block are quantitatively obtained. A scheduling cost calculation system is constructed according to multi-dimensional data characteristics, access requirements, the running state of a storage layer, and compression costs, and the target storage layer and compression mode are optimally matched, and the migration execution order is determined based on the access correlation relationship of the data block. The performance indicators of the layered storage are acquired, the calculation parameters of the scheduling cost are adjusted in reverse, dynamic working conditions are continuously adapted, and the application has strong scene adaptability and long-term running stability.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data scheduling system and method based on a multi-core processor and a multi-layer storage architecture. Background Technology

[0002] With the rapid iteration of multi-core processor architecture and the upgrading of computing power requirements in embedded, server and high-performance computing scenarios, multi-core processors have become the mainstream computing power architecture based on their multi-core parallel computing capabilities. At the same time, in order to make up for the technical shortcomings of the mismatch between processor computing speed and the read / write speed, capacity and power consumption of a single storage medium, a hierarchical combination of high-speed cache, main memory and low-speed large-capacity external storage is used to take into account data access latency, storage capacity and hardware power consumption, and to achieve hierarchical adaptation management of storage resources.

[0003] While existing traditional multi-layer storage scheduling schemes can achieve basic data tiered storage and migration scheduling, they have inherent technical defects in the dynamic adaptability of multi-core collaborative memory access, the completeness of data feature characterization, and the quantitative accuracy of scheduling decisions. They cannot adapt to high-concurrency, highly coupled, and dynamically changing multi-core business conditions. Existing scheduling decisions are based solely on historical static memory access data, which cannot capture the phase switching characteristics and dynamic memory access trends during processor core execution, making it difficult to adapt to the dynamically fluctuating memory access needs of multi-core processors.

[0004] Current methods rely solely on the frequency of single memory accesses to determine hot and cold data characteristics, treating each data block as an independent entity for scheduling decisions. This ignores the cross-core address dependencies, timing sequences, and data flow relationships commonly found in multi-core business operations. Consequently, data characteristics are portrayed in a one-sided manner, and hot / cold data determination is distorted, failing to support differentiated scheduling. Furthermore, current storage layer matching, compression method selection, and migration order arrangement all employ static and fixed rules, lacking a comprehensive optimization mechanism that quantifies costs across multiple dimensions. This commonly results in issues such as high-performance storage resources being occupied by low-value data, misplaced storage of high-affinity core blocks, and mismatches between compression methods and data characteristics, as well as disordered migration of related data disrupting business processes. Overall, this leads to high scheduling overhead, low resource utilization, and poor memory access performance.

[0005] In view of the common shortcomings of existing technologies, the technical problem to be solved by this application is how to achieve accurate, dynamic, and long-term adaptive data scheduling in a multi-core processor and multi-layer storage system. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this application is to provide a data scheduling system and method for a multi-layer storage architecture based on multi-core processors, which improves memory access response performance, storage resource utilization and data scheduling accuracy, and is suitable for high-concurrency and strongly correlated data scheduling scenarios under a multi-layer storage architecture with multi-core processors.

[0007] To achieve the above objectives, this application adopts the following technical solution:

[0008] Firstly, this application provides a data scheduling method for a multi-layered memory architecture based on a multi-core processor, including:

[0009] The memory access event stream of each processor core is obtained, the execution phase changes of the processor core are identified, the access requirements of the corresponding processor core are predicted, and the operating status of each storage layer in the storage system is obtained.

[0010] Address dependency and timing correlation analysis are performed on all memory access event streams. A data flow graph with data blocks as nodes is constructed. Based on the topological characteristics of the data flow graph and the inter-core access distribution and effective access duration of different data blocks, the sharing degree, affinity and lifespan of data blocks are determined.

[0011] Based on the sharing degree, affinity, lifespan and access requirements of data blocks, combined with the operating status of the storage layer and the amount of data to be migrated and the compression cost under the compression method, the scheduling cost is calculated and the target storage layer and corresponding compression method are determined. The migration execution order is determined according to the access relationship of data blocks within the data block group.

[0012] Data blocks are migrated to the target storage layer according to the migration execution order, and compression processing is performed according to the corresponding compression method. Based on the performance indicators obtained after the migration execution, the calculation parameters of the scheduling cost are adjusted.

[0013] Furthermore, the predicted access requirements for the corresponding processor core include:

[0014] The memory access events of the processor core are acquired and concatenated to form a memory access event stream, and the hardware operation data of the processor core within the same sampling period are acquired.

[0015] By comparing the distribution characteristics of memory access events with the numerical changes of hardware operation data within adjacent sampling periods, the execution phase changes of the processor core are identified based on the comparison results.

[0016] Based on the execution phase change, the behavioral characteristics of memory access events under the current execution phase are extracted, and the feature weights are set according to the phase duration to obtain the access status of the processor core.

[0017] The system analyzes data sharing and resource contention behaviors among processor cores, adjusts access patterns based on the statistical results, and predicts the access demands of corresponding processor cores.

[0018] Furthermore, constructing the data flow graph includes:

[0019] It aggregates the memory access event streams from all processor cores and groups the memory access events to the corresponding data blocks according to their physical addresses;

[0020] Analyze the address dependencies and timing relationships of cross-core memory access, filter and retain the access relationships between data blocks to form the association edges between data blocks;

[0021] Using data blocks as nodes, node connections are established based on the parsed address dependencies and temporal sequence relationships, and feature attributes are labeled for the associated edges according to the degree of interaction between the associated edges.

[0022] Integrate the nodes, associated edges, and characteristic attributes of all data blocks to form a data flow graph.

[0023] Furthermore, determining the sharing degree, affinity, and lifespan of data blocks includes:

[0024] Extract the topological features of nodes from the data flow graph, and statistically analyze the inter-core access distribution of data blocks and the effective access duration within the sampling period;

[0025] The sharing degree of a data block is determined by combining the node coverage area and the effective status of associated edges with the number of processor cores accessing the corresponding data block and the effective access duration.

[0026] Based on the valid state of the associated edges, and combined with the access ratio and valid access duration of a single processor core to the data block, the affinity between the processor core and the data block is obtained by comprehensive comparison.

[0027] The system determines the existence status of nodes, combines the distribution activity of data block access with the effective access duration for joint calculation, and determines the life cycle of data blocks based on the calculation results.

[0028] Furthermore, the topological features include node coverage, existence status, and valid status of associated edges; the inter-core access distribution includes the number of processor cores accessing data blocks, the access ratio of a single processor core to a data block, and the distribution activity of data block access; the distribution activity is obtained based on the access frequency statistics of processor cores.

[0029] Based on the access relationships between data blocks, the data blocks are grouped together to determine the data block group.

[0030] Furthermore, determining the target storage layer and corresponding compression method includes:

[0031] Using the sharing degree and lifespan of data blocks as constraints, the access popularity weight of data blocks is determined in combination with access requirements, and the operating status of each storage layer in the storage system is matched to obtain the initial matching result.

[0032] The initial matching results are corrected based on the affinity between the processor core and the data block to obtain candidate storage layers. The amount of migrated data and the compression cost under different compression methods are traversed, and the compression cost is dynamically compensated based on the current load status of the candidate storage layers to obtain the scheduling cost.

[0033] By comparing the scheduling costs of each candidate storage layer with different compression methods, the storage combination with the lowest scheduling cost is selected as the target storage layer and corresponding compression method.

[0034] Furthermore, the access popularity weight is generated by defining the value range of the data block's sharing degree and life cycle, using the value range as the constraint boundary, and combining it with access requirements to perform numerical mapping.

[0035] The scheduling cost is generated by jointly calculating the amount of migrated data under the current compression method and the compensated compression cost; the storage combination includes the storage layer and its corresponding compression method.

[0036] Furthermore, determining the migration execution order includes:

[0037] The migration sequence chain of each group of data blocks is constructed based on the access relationship, and the basic migration order of data blocks within the data block group is determined by the degree of interaction of the associated edges.

[0038] The urgency of data failure is determined by quantifying the lifecycle of data blocks to optimize the basic migration order. The migration order is also adjusted by considering the scheduling cost of data blocks to determine the migration execution order.

[0039] Furthermore, the calculation parameters for adjusting the scheduling cost include:

[0040] The hierarchical storage access latency, multi-core memory access hit rate and compression cost deviation after data block migration and compression are obtained as performance indicators. Based on the performance indicators, the types of deviations such as heat mapping deviation, compression compensation deviation and scheduling operation deviation are identified in reverse.

[0041] Adjust the numerical mapping parameters of the access popularity weight, the compensation parameters of the compression cost, and the joint operation parameters according to the type of deviation.

[0042] During parameter adjustment, the access relationships of data block groups and the criteria for determining their lifecycle are locked.

[0043] Secondly, this application provides a multi-layer storage architecture data scheduling system based on a multi-core processor, including: an acquisition module, an analysis module, a scheduling module, and an execution module;

[0044] The acquisition module is used to acquire the memory access event stream of each processor core, identify the execution phase changes of the processor core, predict the access requirements of the corresponding processor core, and acquire the operating status of each storage layer in the storage system.

[0045] The analysis module is used to perform address dependency and timing correlation analysis on all memory access event streams, construct a data flow graph with data blocks as nodes, determine the sharing degree, affinity and lifespan of data blocks based on the topological characteristics of the data flow graph and the inter-core access distribution and time span of different data blocks, and determine data block groups based on access correlations;

[0046] The scheduling module is used to calculate the scheduling cost and determine the target storage layer and corresponding compression method based on the sharing degree, affinity, life cycle and access requirements of data blocks, combined with the operating status of the storage layer and the amount of data to be migrated and the compression cost under the compression method. It also determines the migration execution order based on the access relationship of data blocks within the data block group.

[0047] The execution module is used to migrate data blocks to the target storage layer according to the migration execution order, perform compression processing according to the corresponding compression method, and adjust the calculation parameters of the scheduling cost based on the performance indicators obtained after the migration execution.

[0048] Compared with the prior art, the beneficial effects achieved by this application are as follows:

[0049] By acquiring multi-core memory access event streams and identifying execution phase changes of processor cores, a technological upgrade from passive scheduling to proactive prediction is achieved. This predicts the access needs of corresponding processor cores, significantly improving the continuity and real-time response performance of multi-core processor memory access links, and adapting to high-concurrency and dynamically fluctuating multi-core parallel business scenarios. By constructing a data flow graph through address dependency and time-series correlation analysis, and based on the topological characteristics, inter-core access distribution, and effective access duration of the data flow graph, the sharing degree, affinity, and lifespan of data blocks are quantified, providing accurate and comprehensive quantitative data support for scheduling.

[0050] Based on multidimensional data characteristics, access requirements, storage layer operating status, and compression costs, a scheduling cost calculation system is constructed to achieve globally optimal quantitative decision-making for storage layers, compression methods, and migration timing. It also selects and matches target storage layers and compression methods optimally, and determines the migration execution order based on the access relationships of data blocks, ensuring the integrity of cross-block data business links and significantly improving the utilization rate of multi-layer storage resources and overall scheduling economy. By obtaining the performance indicators of layered storage, the calculation parameters of scheduling costs are adjusted in reverse to continuously adapt to dynamic working conditions, exhibiting strong scenario adaptability and long-term operational stability. Attached Figure Description

[0051] Figure 1 A flowchart illustrating a data scheduling method for a multi-layered storage architecture based on a multi-core processor;

[0052] Figure 2 A logical flowchart for constructing a data flow diagram;

[0053] Figure 3 This is a schematic diagram of a multi-layered storage architecture data scheduling system based on a multi-core processor. Detailed Implementation

[0054] The technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of this application, rather than limitations thereof.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment provides a data scheduling method for a multi-layer storage system based on a multi-core processor, including: S1, obtaining the memory access event stream of each processor core, identifying the execution phase change of the processor core to predict the access requirements of the corresponding processor core, and obtaining the operating status of each storage layer in the storage system.

[0057] Specifically, the predicted access requirements for the corresponding processor cores include:

[0058] The memory access events of the processor core are acquired and concatenated to form a memory access event stream, and the hardware operation data of the processor core within the same sampling period are acquired.

[0059] By comparing the distribution characteristics of memory access events with the numerical changes of hardware operation data within adjacent sampling periods, the execution phase changes of the processor core are identified based on the comparison results.

[0060] Based on the execution phase change, the behavioral characteristics of memory access events under the current execution phase are extracted, and the feature weights are set according to the phase duration to obtain the access status of the processor core.

[0061] The system analyzes data sharing and resource contention behaviors among processor cores, adjusts access patterns based on the statistical results, and predicts the access demands of corresponding processor cores.

[0062] Processor cores exhibit differentiated memory access behaviors and hardware operating states at different task stages, and single discrete data cannot characterize continuous operating patterns. A uniform sampling period of 10ms is set, which matches the task switching and memory access statistics granularity of mainstream multi-core processors, striking a balance between data integrity and real-time performance. A data acquisition unit is deployed on each processor core in the architecture to add timestamps to single read and write memory access events generated during operation. Discrete memory access events are linearly concatenated according to their chronological order to form a time-series continuous memory access event stream, thereby reconstructing continuous access behavior.

[0063] Within each 10ms sampling period, the hardware operation data of the processor core is acquired synchronously, including the instantaneous load rate, instruction throughput and on-chip cache hit rate of the processor core. The memory access event stream generated in the same period is time-bound and stored with the hardware operation data to build a single-core operation dataset, ensuring that the time base of behavioral data and status data is consistent, and avoiding feature misalignment problems caused by asynchronous acquisition from the source.

[0064] The aforementioned data acquisition and binding operations provide the original data foundation for subsequent cross-cycle feature comparison and phase recognition. The temporal integrity and indicator accuracy of the data determine the reliability of the subsequent phase determination results.

[0065] After completing the construction of the periodic dataset, the execution phase change is identified by comparing features of adjacent periods. The execution phase refers to the stable operating phase when the processor core is executing a specific business task. Under the same phase, the memory access distribution pattern and hardware status indicators will maintain a relatively fixed range. The phase switching is accompanied by a continuous shift in feature data. Specifically, the memory access event stream within a single sampling period is first extracted as a feature. The three items of memory access address interval concentration, read / write event ratio, and memory access event density per unit time are quantified into memory access distribution feature vectors. At the same time, the processor core load rate, instruction throughput, and cache hit rate are integrated into hardware status feature vectors. The two types of vectors together constitute a complete feature set for a single period.

[0066] This embodiment uses Euclidean distance to calculate the similarity of adjacent period feature vectors, thereby quantifying the feature fluctuation amplitude. The feature similarity threshold is set to 0.85, which is determined based on the normal operating fluctuation range of a multi-core processor. The feature vectors of the Nth and N+1th sampling periods are read sequentially and substituted into the similarity calculation logic to complete the operation. If the calculated similarity is greater than or equal to 0.85, it indicates that there is no significant shift in the features of adjacent periods, and the processor core is determined to maintain its original execution phase. If the similarity is less than 0.85, it indicates that there is a continuous change in memory access behavior and hardware state, and the processor core is determined to have switched execution phases. The new execution phase type is defined based on the distribution range of the shifted feature vectors.

[0067] By using feature vector quantization and similarity calculation, the automatic identification of execution phase changes is achieved, improving the objectivity and accuracy of state recognition. The output execution phase type and phase switching determination results serve as a prerequisite for subsequent extraction of memory access behavior features, affecting the targeting of behavior feature extraction.

[0068] Based on the current execution phase, memory access behavior features of the phase are extracted. The features are weighted and fused together with the phase duration to finally generate the access status of the processor core. Different execution phases correspond to memory access behavior patterns. The phase duration can reflect the stability of the current running state. The behavior features of steady-state phases have higher reference value, while the behavior of transition phases is more random. Specifically, the entire periodic dataset corresponding to the current execution phase is extracted first, and three types of behavioral features are extracted from it: memory access address preference, batch memory access granularity, and continuous memory access timing pattern. Each type of feature is quantified and the numerical range is unified.

[0069] The duration of the current execution phase is statistically analyzed in real time. Three duration intervals are defined by two boundary nodes: 50ms and 200ms. Corresponding feature weights are then matched accordingly. Phases with a duration less than 50ms are classified as transitional phases with a weight coefficient of 0.3; phases with a duration between 50ms and 200ms are classified as metastable phases with a weight coefficient of 0.6; and phases with a duration greater than 200ms are classified as steady-state phases with a weight coefficient of 0.9. A linear weighted summation method is used to integrate multiple features. This involves multiplying the quantized value of each feature by its corresponding weight coefficient. The sum of all results gives the access pattern, representing the overall memory access tendency of the processor core. For example, if the quantized values ​​of the three behavioral features in a steady-state phase are 72, 65, and 80, with each corresponding to a weight of 0.9, the resulting access pattern is (72×0.9)+(65×0.9)+(80×0.9)=195.3.

[0070] The aforementioned weighted fusion method distinguishes the reference priority of features under different phase states, allowing the access situation to objectively reflect the inherent access patterns of a single core. The generated access situation is the basic carrier for correcting inter-core interaction behavior. The completeness and quantification accuracy of the situation representation affect the deviation range of the final access demand prediction.

[0071] In multi-core processor scenarios, the memory access behavior of a single processor core is directly affected by the interaction behavior of surrounding cores. Data sharing will increase the access frequency and priority of the target data block, while the contention for hardware and storage resources will suppress the memory access behavior corresponding to conflicting resources. The two types of interaction behaviors have significantly different directions and strengths of influence on the original access situation. The basic access situation generated by relying solely on single-core behavior cannot reflect the real operating conditions.

[0072] Specifically, the first step is to quantify and standardize the inter-core interaction behavior, which forms the data foundation for situational correction. Then, the operation logs and interaction records of all processor cores are traversed to statistically analyze the core interaction metrics of the target processor core. For data sharing behavior, the statistics are cross-core data access frequency and shared data block size; for resource contention behavior, the statistics are bus preemption count and on-chip cache resource conflict duration. All interaction metrics are then normalized and mapped to a range of 0 to 1. Values ​​closer to 1 indicate a higher intensity of the corresponding interaction behavior, while values ​​closer to 0 indicate a weaker interaction behavior.

[0073] For example, if the target processor core has high-frequency cross-core data reads with other cores, the normalized result of its cross-core data access frequency is 0.76, and the normalized result of the shared data block size is 0.68; if there is moderate bus and cache resource contention at the same time, the normalized result of the bus preemption count is 0.42, and the normalized result of the cache conflict duration is 0.39; after completing the normalization of single-category indicators, the comprehensive data sharing indicator and the comprehensive resource contention indicator are calculated respectively. The calculation method is the arithmetic mean of all normalized indicators under the corresponding category, thus obtaining the quantitative values ​​of the two types of interaction behaviors. Using the above example values, the comprehensive data sharing indicator is (0.76+0.68)÷2=0.72, and the comprehensive resource contention indicator is (0.42+0.39)÷2=0.405.

[0074] The aforementioned basic access situation is generated by weighted fusion of memory access address preference, batch memory access granularity, and continuous memory access timing pattern. The situation correction adopts a sub-item correction mechanism, that is, each original memory access behavior feature is calculated separately, distinguishing the correction logic for data sharing and resource contention behaviors. For data sharing behavior, this behavior will increase the memory access tendency corresponding to shared data, so a positive correction coefficient is set. The coefficient value range matches the sharing comprehensive index range. The higher the sharing comprehensive index, the larger the positive correction coefficient value. In this embodiment, the value range of the sharing correction coefficient is set to 0 to 0.3. For resource contention behavior, this behavior will reduce the memory access tendency corresponding to conflicting resources, so a negative correction coefficient is set. The coefficient value range is -0.3 to 0. The higher the resource contention comprehensive index, the larger the absolute value of the negative correction coefficient.

[0075] For example, the data sharing comprehensive index of 0.72 corresponds to a positive correction coefficient of 0.22, and the resource contention comprehensive index of 0.405 corresponds to a negative correction coefficient of -0.13. The correction operation of the sub-features is based on the original feature quantization value. The operation rule is that the corrected feature value is equal to the original feature quantization value plus the product of the original feature quantization value and the two correction coefficients. The three behavioral features under the aforementioned steady-state phase are 72, 65, and 80 respectively. Substituting them into the calculation, the corrected value of the first feature is 72 + 72 × 0.22 + 72 × (-0.13), the corrected value of the second feature is 65 + 65 × 0.22 + 65 × (-0.13), and the corrected value of the third feature is 80 + 80 × 0.22 + 80 × (-0.13). The calculated corrected feature values ​​of the three features are 78.48, 70.85, and 87.2 respectively. This is to accurately distinguish the impact of different interaction behaviors on single-type memory access features and avoid feature distortion caused by overall general correction.

[0076] After completing the sub-item memory access feature correction, the corrected access status is reconstructed using the aforementioned linear weighted summation, keeping the feature weights corresponding to the phase unchanged to ensure the consistency of the phase determination benchmark. In the previous text, this phase was a steady-state phase with a feature weight of 0.9. The three corrected feature values ​​are multiplied by their corresponding weights in sequence and then summed. The corrected access status is equal to the sum of the corrected feature values ​​multiplied by their corresponding weights. Substituting the corrected feature values ​​into the above calculation, the corrected access status is 78.48×0.9+70.85×0.9+87.2×0.9=212.877. This result integrates the inherent memory access rules of a single core, the execution phase characteristics, and the influence of multi-core interaction behavior, more objectively reflecting the access status of the processor core under multi-core collaborative working conditions.

[0077] Based on the corrected access status and the sub-item memory access characteristics, a hierarchical prediction of processor core access requirements is completed. The maximum-minimum range normalization algorithm is adopted. The global maximum and global minimum values ​​are two boundary values ​​extracted after traversing the original values ​​of the access status corresponding to all processor cores and all execution phases in the system. The normalized access status is equal to the current sample original value minus the global minimum value, and then divided by the difference between the global maximum and global minimum values. The basic access status original value under a certain steady-state phase is 195.3. Assuming that the global basic status minimum is 40 and the global basic status maximum is 220, the normalized value of this set of access statuses can be obtained by substituting them. The numerical range of the normalized access status is pre-divided, and different ranges correspond to different access intensity levels. At the same time, the access target and access operation type are defined by combining the three sub-item memory access characteristics.

[0078] First, the overall access intensity is determined based on the normalized access pattern. If the value is in the high range of 0.7~1, it is determined to be a high-intensity continuous access demand; if it is in the middle range of 0.35~0.7, it is determined to be a regular intermittent access demand; if it is in the low range of 0~0.35, it is determined to be a low-frequency sporadic access demand. Next, three corrected sub-features are read: memory access address preference feature is used to define the target data block range of subsequent accesses, batch memory access granularity feature is used to define the data volume of a single access, and continuous memory access timing pattern feature is used to define the arrangement of read and write operations. Combining the normalized access pattern in this example, it is determined that the processor core has a high-intensity continuous access demand. Then, combined with sub-features such as address preference and memory access granularity, the target access data block, the data size of a single access, and the timing of read and write operations are locked, and finally, an access demand prediction result including access intensity, access object, operation type, and access rhythm is formed.

[0079] Based on the quantitative correction of inter-core interaction behavior, the different logics of data sharing and resource contention on memory access behavior are accurately distinguished. The original phase characteristics and memory access patterns are preserved through the correction of sub-features, while the behavioral deviations caused by multi-core interaction are offset, so that the representation of access status is consistent with the actual operating state of multi-core processors. The access requirements of processor cores are the core input data of the subsequent full-process scheduling.

[0080] S2. Perform address dependency and timing correlation analysis on all memory access event streams, construct a data flow graph with data blocks as nodes, determine the sharing degree, affinity and lifespan of data blocks based on the topological characteristics of the data flow graph and the inter-core access distribution and effective access duration of different data blocks, and determine the data block group based on the access correlation relationship.

[0081] Furthermore, such as Figure 2 As shown, constructing a data flow diagram includes:

[0082] It aggregates the memory access event streams from all processor cores and groups the memory access events to the corresponding data blocks according to their physical addresses;

[0083] Analyze the address dependencies and timing relationships of cross-core memory access, filter and retain the access relationships between data blocks to form the association edges between data blocks;

[0084] Using data blocks as nodes, node connections are established based on the parsed address dependencies and temporal sequence relationships, and feature attributes are labeled for the associated edges according to the degree of interaction between the associated edges.

[0085] Integrate the nodes, associated edges, and characteristic attributes of all data blocks to form a data flow graph.

[0086] The memory access event streams acquired by multi-core processors are isolated from each other and can only characterize the timing memory access behavior of a single core. There is no statistical benchmark for data dimensions, so it cannot be directly used to explore data relationships across cores and blocks. The memory access event stream is a continuous read and write event sequence formed by sequentially splicing each processor core according to a unified sampling period and timestamp. The physical address is the linear addressing identifier of the storage hardware, and the data block is a preset fixed-granularity storage data unit. Specifically, the hardware data block division granularity is set to 4KB physical page size, and the hardware fixed physical address range is used as the matching basis. A static mapping table between the global physical address range and the data block number is generated in advance.

[0087] It aggregates the full-time memory access event streams of all processor cores, breaking down the data isolation boundaries of a single core and integrating them into a memory access event dataset. For each memory access event in the dataset, it extracts the physical address it carries, retrieves the static mapping table to determine the data block number to which it belongs, and completely collects all types of memory access events, such as read memory access, write memory access, and batch memory access, to the corresponding data block. For multiple memory access behaviors of the same data block under different sampling periods and different processor cores, all are retained and archived, completely preserving the global access trajectory and frequency characteristics of the data block. Using hardware physical attributes as the judgment criterion, it achieves accurate conversion of discrete time-series memory access events to the structured data block dimension and unifies the statistical caliber of memory access behavior.

[0088] The above method eliminates the technical defects of multi-core data isolation and discrete event fragmentation, providing a regular, complete and unbiased basic data sample for subsequent cross-block association resolution; the data block is the preceding data source for subsequent resolution of address dependency, temporal sequence and construction of associated edges, and determines the authenticity of the data flow graph topology.

[0089] Access to a single data block does not have topological association value; address dependency is the business binding relationship between different data blocks at the addressing logic, data read / write and index mapping level; temporal sequence relationship is the temporal constraint relationship of the sequential triggering and successive execution of memory access events of different data blocks in the task execution chain; the association edge is the topological connection unit used to carry the effective association relationship between two data blocks; specifically, all cross-processor core-initiated cross-data block memory access behaviors are selected from the globally collected data as parsing samples, and data block access behaviors within a single core are eliminated.

[0090] A two-layer association determination is performed on the sample data. At the address dependency level, three types of verifiable determination rules are set: data index addressing dependency, physical address offset association dependency, and data overwrite linkage dependency. If any one of these rules is met, a valid address dependency can be determined. At the timing continuity level, a fixed timing association window is set. For example, the timing association window is set to 10ms. If two cross-block memory access events belong to the same business task chain and the event trigger time difference falls within the 10ms window range, a timing continuity relationship can be determined.

[0091] Hard association filtering conditions are set up to determine that only cross-block memory access behaviors that simultaneously satisfy address dependency and time sequence relationship are judged as valid access associations. Discrete random access that exceeds the time sequence window, occasional cross-core access without business link support, and independent read and write behaviors without address binding are all judged as invalid noise associations and completely eliminated. All the filtered valid associations are recorded in a structured way, and the original association edges between each pair of data blocks are initialized. Only the topology connections with business linkage value are retained.

[0092] By using a two-layer binding mechanism to accurately filter out invalid memory access noise, it ensures that all associated edges truly map the actual data flow patterns of multi-core business operations. The selected valid relationships and edges lock in the topological connection architecture of the data flow graph, providing an effective framework for subsequent node connection construction and edge attribute quantification labeling.

[0093] The initially generated associated edges only have topological connectivity and no strength distinction, which cannot quantitatively represent the differences in the depth of interaction coupling between different data blocks, and cannot support the subsequent fine-grained quantitative calculation of sharing degree and affinity. The degree of interaction tightness is a quantitative index that comprehensively represents the frequency, duration and temporal coupling of the interaction between two data blocks. The feature attribute is a set of quantitative parameters attached to the associated edges, which is used to record the interaction characteristics of the bilateral data. Specifically, each data block collected in the first step is a topological node. Based on the effective address dependency and temporal continuity relationship obtained in the second step, an undirected topological connection is established between two sets of nodes with effective association, forming an initial global topological network.

[0094] For each valid associated edge, the cross-core memory access frequency, continuous business interaction duration, and time-series matching overlap are obtained. The three types of original parameters are normalized to the range of 0 to 1. Using equal-weighted fusion logic, the arithmetic mean of the three normalized parameters is calculated to obtain the interaction tightness of the associated edge. For example, if the normalized interaction frequency, interaction duration, and time-series overlap of an associated edge are 0.82, 0.75, and 0.79, respectively, the interaction tightness is calculated to be 0.787. The three types of original interaction parameters, the normalization result, and the final interaction tightness are encapsulated as feature attributes of the associated edge and bound to the corresponding topological edge to achieve deep binding between topological connection relationship and interaction quantification features.

[0095] The above steps upgrade the data flow graph from a simple structural topology to a vectorized topology, enabling each topological edge to have comparable, computable, and statistically significant quantitative characteristics, accurately distinguishing the business coupling strength of different data blocks. The associated edges are the core calculation basis for subsequent extraction of topological features, statistical distribution of inter-core access, calculation of data block sharing degree and core-block affinity, and determine the refinement and accuracy of the subsequent quantification of all data indicators.

[0096] Discrete and independent topological nodes, associated edges, and feature parameters cannot form a globally unified structured model, lacking the capabilities for global feature retrieval, batch statistics, and topological feature extraction. Integration mapping involves unified verification, regularization, and structured binding of global topological units to form a closed-loop connected topology. Specifically, it traverses all data block topological nodes and all associated edges globally, completing a three-layer integration mapping operation. The first layer performs topological deduplication verification, eliminating duplicate established edges with the same origin, ensuring that only one edge is retained between any two sets of data blocks. The second layer performs invalid topology cleanup, eliminating dangling isolated nodes without any memory access records or feature attribute assignments, as well as invalid dangling edges, ensuring the validity of the topological network.

[0097] The third layer performs global binding mapping, binding the physical address identifier, storage level location, and full memory access statistics of each node with the corresponding node topology information. At the same time, the feature attribute dataset of each associated edge is synchronously entered into the global topology framework to complete the accurate mapping of nodes, connections, and quantization attributes. After global regularization and integration, a multi-core storage data flow graph is finally generated with data blocks as nodes, effective cross-block access associations as topology edges, and interactive tight quantization parameters as edge features.

[0098] The above steps construct a complete, closed-loop, quantifiable, and analyzable global topology, realizing the complete transformation from multi-core discrete memory access events to structured data flow topology, and fully restoring the data dependency and interaction coupling rules of multi-core storage. The data flow graph is the core basic carrier for extracting topological features, statistically analyzing inter-core access distribution, calculating lifecycles, and aggregating data block groups, providing the underlying topological support for the subsequent full-process scheduling logic of heat weight calculation, scheduling cost compensation, and migration order sorting.

[0099] Specifically, determining the sharing degree, affinity, and lifespan of data blocks includes:

[0100] Extract the topological features of nodes from the data flow graph, and statistically analyze the inter-core access distribution of data blocks and the effective access duration within the sampling period;

[0101] The sharing degree of a data block is determined by combining the node coverage area and the effective status of associated edges with the number of processor cores accessing the corresponding data block and the effective access duration.

[0102] Based on the valid state of the associated edges, and combined with the access ratio and valid access duration of a single processor core to the data block, the affinity between the processor core and the data block is obtained by comprehensive comparison.

[0103] The system determines the existence status of nodes, combines the distribution activity of data block access with the effective access duration for joint calculation, and determines the life cycle of data blocks based on the calculation results.

[0104] The topological features include node coverage, existence status, and the valid status of associated edges; the inter-core access distribution includes the number of processor cores accessing data blocks, the proportion of access to data blocks by a single processor core, and the distribution activity of data block access; the distribution activity is obtained by statistically analyzing the access frequency of processor cores.

[0105] Based on the access relationships between data blocks, the data blocks are grouped together to determine the data block group.

[0106] An unorganized data flow graph only has topological connections and cannot support multi-indicator fusion calculations. Topological features include node coverage, node persistence status, and the effective status of associated edges. Inter-core access distribution includes the number of accessing cores, the percentage of single-core accesses, and access distribution activity, where distribution activity is generated by memory access frequency statistics. Specifically, full data extraction and statistics are carried out based on the data flow graph, with a global minimum sampling and statistics period of 10ms. Node coverage is the total number of adjacent nodes corresponding to the effective associated edges directly connected to the target data block node, representing the radiation scale of the data block in the global topology.

[0107] The persistence status of a node is the number of sampling periods in which the data block node continuously exists in the data flow graph, used to distinguish between temporary instantaneous data nodes and steady-state business data nodes; the valid status of an associated edge refers to the fact that only associated edges that pass the dual verification of a 10ms timing window and address dependency are marked as valid edges, while other randomly accessed associated edges are marked as invalid edges; the memory access logs within multiple consecutive sampling periods are traversed to count the total number of processor cores that initiate the access behavior of the target data block; for each processor core participating in the access, the ratio of the number of single-core memory accesses to the total number of memory accesses for that data block is calculated to obtain the single-core access ratio.

[0108] The original memory access frequencies of different data blocks are uniformly mapped to the range of 0 to 1. That is, the distribution activity of a single data block = (current data block memory access frequency - global minimum memory access frequency) / (global maximum memory access frequency - global minimum memory access frequency). The closer the value is to 1, the more intensive and active the access behavior is. For example, if the global minimum memory access frequency is 2 times / 10ms, the global maximum memory access frequency is 20 times / 10ms, and the memory access frequency of a certain target data block is 11 times / 10ms, the distribution activity of the data block is calculated to be 0.5. At the same time, idle periods, memory access error periods, and invalid random access periods are removed. Only the effective continuous duration of effective read and write interactions and business data flow of data blocks is accumulated to ensure that the statistical data fully matches the actual business conditions.

[0109] The above steps complete the standardization, normalization, and quantization of all original parameters in the topology and memory access dimensions, eliminating dimensional differences and providing a compliant and unified computing basis for subsequent multi-parameter fusion operations. The accuracy of parameter statistics determines the authenticity and effectiveness of all subsequent quantitative indicators.

[0110] Sharing degree is used to quantify the extent to which data blocks are collaboratively reused by multiple cores and shared globally by business processes. A single topology parameter or a single memory access parameter may have limitations in judgment. Specifically, all valid associated edges of the target data block are first screened, invalid and noisy associated edges are removed, and the total number of valid associated edges and the number of corresponding adjacent nodes are counted to quantify the node topology coverage scale. Then, the total number of accessing cores, normalized distribution activity, and effective access duration corresponding to the data block are retrieved. In this embodiment, an equal-weighted fusion mechanism is used to realize the quantitative calculation of sharing degree. The four parameters of topology coverage scale, percentage of valid associated edges, number of multi-core accesses, and effective access duration are normalized to the interval of 0 to 1 and then the arithmetic mean is calculated to generate the sharing degree.

[0111] For example, a business data block has a normalized topology coverage of 0.81, an effective associated edge ratio of 0.76, a multi-core access count of 0.79, and an effective access duration of 0.84. After the four parameters are combined and calculated, the data block has a sharing degree of 0.8, which is a high-sharing data block. On the other hand, a local temporary data block has four parameters of 0.21, 0.18, 0.25, and 0.22, respectively. After merging, the sharing degree is 0.21, which is a low-sharing data block. This achieves bidirectional coupling judgment between topology structure and actual memory access behavior, avoiding the bias problem of single-dimensional judgment.

[0112] The above steps accurately quantify and distinguish the global sharing and reuse capabilities of different data blocks, providing a quantitative basis for subsequent hot and cold data determination and storage layer adaptation; the sharing degree is the core constraint parameter for the access popularity weight mapping and initial storage layer matching, which determines the storage scheduling priority of data blocks.

[0113] Affinity is used to quantitatively characterize the access coupling and binding relationship between a single processor core and a target data block, and is the core quantitative basis for multi-core differentiated scheduling. Specifically, all valid associated edges of the target data block are first locked to confirm that there is a real business data flow association between the processor core and the data block, and invalid statistical data without topological association are excluded. For each core-block combination, two core parameters, single-core access ratio and single-core effective access duration, are retrieved and normalized to the interval of 0 to 1 respectively. Affinity is calculated by dual-parameter weighted fusion, with a single-core access ratio weight of 0.5 and a single-core effective access duration weight of 0.5. The affinity is generated by weighted summation, that is, affinity = normalized access ratio × 0.5 + normalized effective access duration × 0.5.

[0114] For example, if a processor core has a normalized access percentage of 0.86 and a normalized effective access duration of 0.82, the affinity of this core-block relationship is 0.84, indicating a strong affinity. Another processor core has a normalized access percentage of 0.23 and a normalized access duration of 0.19, resulting in an affinity of 0.21, indicating a weak affinity. This accurately distinguishes the degree of access dependency of different processor cores on the same data block.

[0115] The above steps construct a refined kernel-block coupling quantization system, breaking the defect of indiscriminate adaptation in traditional storage scheduling, and providing a precise basis for subsequent storage layer matching result correction; affinity is the core parameter for correcting the initial storage matching result and optimizing the accuracy of scheduling cost, and determines the suitability of the target storage combination.

[0116] Lifetime is used to characterize the effective business continuity capability, access decay trend, and urgency of data failure of data blocks. Specifically, the topology stability level is first divided based on the node's survival status, with the number of sampling periods during which data block nodes continuously exist serving as the criterion. The more sampling periods during which nodes survive, the higher the topology stability level. Based on this, the normalized distribution activity and standardized effective access duration parameters are retrieved, and the weights of topology survival level, distribution activity, and effective access duration are set to 0.3, 0.4, and 0.3 respectively. Lifetime = Normalized topology survival value × 0.3 + Normalized distribution activity × 0.4 + Normalized effective access duration value × 0.3.

[0117] For example, a steady-state business data block has a topology persistence normalization value of 0.85, a distribution activity of 0.8, and a valid access duration normalization value of 0.83. Substituting these values, the lifespan is calculated to be 0.82, thus classifying it as a long-term steady-state valid data block. A temporary business data block has a topology persistence normalization value of 0.22, a distribution activity of 0.25, and a valid access duration normalization value of 0.2. The calculated lifespan is 0.22, thus classifying it as a short-term easily failed data block. Lifespan levels are classified based on the quantified value range to distinguish between long-term steady-state data, medium-term intermittent data, and short-term temporary data.

[0118] The above steps enable dynamic quantitative determination of the data block lifecycle, taking into account both topological stability and actual memory access behavior characteristics, and avoiding the single defect of traditional methods that rely solely on access frequency to determine hot and cold data. The lifecycle is the core constraint parameter for subsequent access heat weight mapping, scheduling cost calculation and migration order optimization, which determines the migration urgency and scheduling strategy of the data block.

[0119] Discretely scheduled data blocks can disrupt cross-block address dependencies and timing relationships. Specifically, the effective associated edges of the data flow graph are used as the grouping benchmark, and the degree of interaction between associated edges is used as the quantization and aggregation threshold. In this embodiment, 0.5 is set as the grouping determination threshold. If the quantization value of the degree of interaction between the associated edges between any two data blocks is greater than 0.5, it is determined to be a strong business association, and the two types of data blocks are aggregated into the same data block group. If the degree of interaction between the associated edges is less than or equal to 0.5, it is determined to be a weak association or no association, and cross-block aggregation is not performed.

[0120] Traverse all data blocks globally, perform chain-like clustering, and integrate multiple strongly correlated data blocks into a data block group; for data blocks without any valid strong correlations, divide them into a single data block group. All grouping operations preserve the address dependencies and temporal sequences of the original data flow graph and do not disrupt the original topological relationship structure. For example, if the correlation between data blocks A and B is 0.78 and the correlation between data blocks B and C is 0.72, both of which are higher than the grouping decision threshold, A, B, and C are chain-like clustered into the same data block group; all correlation edges of data block D are lower than the grouping decision threshold, so it forms a separate data block group.

[0121] The above steps achieve global data block clustering and grouping, ensuring strong correlation within groups and weak correlation between groups, adapting to the overall scheduling requirements of related data; the data block group is the core foundation for arranging the migration execution order based on the access relationship within the group, determining the rationality of subsequent migration sequence chain construction and migration order optimization.

[0122] S3. Based on the sharing degree, affinity, life cycle and access requirements of data blocks, combined with the operating status of the storage layer and the amount of data to be migrated and the compression cost under the compression method, calculate the scheduling cost and determine the target storage layer and corresponding compression method. Determine the migration execution order based on the access association relationship of data blocks within the data block group.

[0123] Further, determining the target storage layer and corresponding compression method includes:

[0124] Using the sharing degree and lifespan of data blocks as constraints, the access popularity weight of data blocks is determined in combination with access requirements, and the operating status of each storage layer in the storage system is matched to obtain the initial matching result.

[0125] The initial matching results are corrected based on the affinity between the processor core and the data block to obtain candidate storage layers. The amount of migrated data and the compression cost under different compression methods are traversed, and the compression cost is dynamically compensated based on the current load status of the candidate storage layers to obtain the scheduling cost.

[0126] By comparing the scheduling costs of each candidate storage layer with different compression methods, the storage combination with the lowest scheduling cost is selected as the target storage layer and corresponding compression method.

[0127] Among them, the access popularity weight is generated by defining the value range of the data block's sharing degree and life cycle, using the value range as the constraint boundary, and combining the access requirements to generate a numerical mapping.

[0128] The scheduling cost is generated by jointly calculating the amount of migrated data under the current compression method and the compensated compression cost; the storage combination includes the storage layer and its corresponding compression method.

[0129] The sharing degree of data blocks reflects their global reuse capability in a multi-core architecture, while the lifespan characterizes the inherent laws of effective data preservation and access decay. These two indicators together constitute the static attribute boundary of the data. The access requirements of the processor cores reflect the real-time memory access behavior trend. Relying on a single-dimensional parameter alone cannot objectively define the access popularity level of data. The sharing degree is pre-divided into three levels: low sharing interval (0 to 0.35), medium sharing interval (0.35 to 0.7), and high sharing interval (0.7 to 1). Simultaneously, the lifespan is divided into three levels: short lifespan interval (0 to 0.35), medium lifespan interval (0.35 to 0.7), and long lifespan interval (0.7 to 1). The interval boundaries remain fixed throughout the operation, serving as the basic constraint condition for heat mapping.

[0130] For any data block to be scheduled, first retrieve the sharing degree and lifespan of the data block, determine the interval range to which each indicator belongs, and match the corresponding level coefficient according to the interval level: 0.3 for low interval, 0.6 for medium interval, and 0.9 for high interval. After obtaining the interval level coefficients of the two indicators, they are merged according to the rule of balanced weight distribution of static attributes. The weight ratio of the sharing degree interval level coefficient and the lifespan interval level coefficient is set to 0.5, thus forming the basic heat coefficient.

[0131] Based on this, the access demand of the processor core is introduced as a dynamic correction item. The correction coefficient is divided according to the previously predicted access demand type: high-intensity continuous access demand corresponds to a correction coefficient of 1, regular intermittent access demand corresponds to a correction coefficient of 0.7, and low-frequency sporadic access demand corresponds to a correction coefficient of 0.4. The access heat weight = basic heat coefficient × access demand correction coefficient, thus generating the access heat weight.

[0132] For example, a core business data block has a sharing degree of 0.82, falling into the high sharing range with a corresponding range level coefficient of 0.9, and a lifespan of 0.85, falling into the long lifespan range with a corresponding range level coefficient of 0.9. The processor core corresponding to this data block exhibits high-intensity continuous access demand, so the correction coefficient is set to 1. Substituting these values, the access heat weight is (0.9×0.5+0.9×0.5)×1=0.9. A temporary interactive data block has a sharing degree of 0.28 and a lifespan of 0.30, both of which fall into the low range with a range level coefficient of 0.3. The corresponding processor core exhibits low-frequency sporadic access demand, so the correction coefficient is set to 0.4. After calculation, the access heat weight is (0.3×0.5+0.3×0.5)×0.4=0.12.

[0133] After the access popularity weights are generated, the operating status of each storage layer in the multi-layer storage system is polled in real time to obtain the real-time load rate, remaining read / write bandwidth, available storage space, and average access latency of the storage layer. A matching logic between the popularity weights and the operating status of the storage layer is established. High access popularity weights are matched with high-speed storage layers with higher requirements for access latency and bandwidth, medium access popularity weights are matched with mid-level storage layers that balance performance and capacity, and low access popularity weights are matched with low-speed storage layers with large capacity and low access performance requirements. An initial matching degree in the range of 0 to 1 is calculated for each storage layer. The higher the value, the stronger the adaptability of the storage layer to the current data block. The matching degrees of the storage layers are summarized to form the initial matching result.

[0134] By unifying the criteria for judging static attributes of data through interval constraints, the access popularity weight fully reflects the long-term reuse value and short-term access trend of data. The initial adaptation is completed based on the hardware operation status of the storage layer, providing a basic matching sample for subsequent differentiated correction. The initial matching degree set of each storage layer is the data source for subsequent result correction based on affinity, affecting the selection range of candidate storage layers and the overall adaptation effect.

[0135] The initial matching results generated previously only focused on matching data access frequency and storage layer hardware status, without considering the long-term access coupling relationship between processor cores and data blocks in multi-core architectures. This could lead to the problem that the storage layer hardware performance meets the standards, but its compatibility with the access core link is insufficient. At the same time, different compression algorithms have significant differences in data compression ratio, computing resource consumption, and execution time. The compression cost is not a fixed constant, and changes in the real-time load of the storage layer will change the actual cost of compression operations. The static cost calculation method cannot reflect the real operating conditions.

[0136] First, retrieve the normalized affinity score, which ranges from 0 to 1. That is, the corrected matching score = initial matching score × (0.8 + affinity score × 0.2). Set the affinity score as a positive correction factor. When the affinity score is high, the matching score of the corresponding storage layer is improved, strengthening the adaptation priority. When the affinity score is low, the matching score is appropriately reduced, weakening the adaptation priority, thereby reflecting the multi-core differentiated access characteristics. Set 0.6 as the candidate storage layer screening threshold. Iterate through the corrected matching scores of all storage layers. Storage layers with a value greater than or equal to 0.6 are defined as candidate storage layers. Storage layers with poor adaptability are eliminated, thus completing the convergence of the candidate set.

[0137] After determining the candidate storage layer, the various preset data compression methods are iterated through. For each compression method, the original migration data volume and the basic compression cost are calculated. Both parameters are normalized to 0 to 1 according to a unified rule. In order to achieve dynamic adaptive adjustment of the compression cost, the real-time load rate of the candidate storage layer is obtained. The load rate ranges from 0 to 1. The load compensation coefficient = 1 + real-time load rate × 0.3. The higher the storage layer load, the larger the compensation coefficient, and the higher the additional resource overhead generated by the compression operation. Based on this, the compensation cost is calculated, that is, the compensation cost = basic compression cost × load compensation coefficient.

[0138] For example, if the current real-time load rate of a candidate storage layer is 0.7, the load compensation coefficient is 1 + 0.7 × 0.3 = 1.21. If one compression method is selected, its normalized compression cost is 0.33. After dynamic load compensation, the compression cost is updated to 0.33 × 1.21 = 0.3993. The scheduling cost = (normalized migration data volume + compensated compression cost) ÷ 2. The corresponding normalized migration data volume is 0.42, and the scheduling cost is (0.42 + 0.3993) ÷ 2 = 0.4096. For each candidate storage layer and each combination of available compression methods, the above statistical, compensation, and joint operation process is repeated to generate the scheduling cost corresponding to each combination.

[0139] The above steps refine the initial matching results, addressing the issue that traditional storage matching does not distinguish between multi-core access relationships. This ensures that the compression cost aligns with the real-time operating status of the storage layer, achieving standardized integration of various overhead metrics. The scheduling cost corresponding to each group of candidate storage layers and compression method combinations serves as the basis for subsequent optimal solution selection, determining the rationality of the final storage combination selection.

[0140] The multi-layered storage system features a combination of multiple levels and compression algorithms. Different combinations vary in data migration overhead, compression computation overhead, and resource consumption. Specifically, it iterates through all selected candidate storage layers, exhaustively exploring each compression method that can be executed on the corresponding storage layer, enumerating all legal storage combinations, and performing a global horizontal comparison of the scheduling costs of all combinations. Finally, it executes a single-criteria selection logic, directly selecting the storage combination with the lowest scheduling cost as the final execution plan.

[0141] For example, the current data block forms three effective storage combinations. The scheduling cost of the first combination is 0.4096, the scheduling cost of the second combination is 0.5721, and the scheduling cost of the third combination is 0.4835. After numerical comparison, the scheduling cost of the first combination is the minimum. The storage layer in this combination is determined as the target storage layer, and the compression method corresponding to the combination is determined as the final compression method to be executed.

[0142] The above steps minimize the overall scheduling overhead while ensuring data access performance and multi-core compatibility. The target storage layer and compression method guide the subsequent data block migration sequence, data migration operations, and compression processing operations, providing operating data and feedback basis for the iterative adjustment of scheduling cost calculation parameters.

[0143] Specifically, determining the migration execution order includes:

[0144] The migration sequence chain of each group of data blocks is constructed based on the access relationship, and the basic migration order of data blocks within the data block group is determined by the degree of interaction of the associated edges.

[0145] The urgency of data failure is determined by quantifying the lifecycle of data blocks to optimize the basic migration order. The migration order is also adjusted by considering the scheduling cost of data blocks to determine the migration execution order.

[0146] Data block groups are generated by clustering related data blocks with address dependencies and temporal sequences. Each data block in a group has a fixed sequence of memory accesses and data flow dependencies in the business execution chain. If migration operations are carried out in an unordered manner without deviating from the original access relationships, it will disrupt the topological coupling relationship between data blocks, causing technical problems such as data read / write chain breaks, memory access sequence disorder, and data consistency anomalies. Access relationships refer to the effective topological relationships in the aforementioned data flow diagram that have been verified by both address dependencies and temporal sequences, including the binding relationships of data read / write dependencies, temporal sequence execution, and business chain linkage between data blocks.

[0147] The migration sequence chain refers to a chain-like time structure constructed with data blocks within a group as nodes, the original business access sequence as constraints, and effective associated edges as connection carriers. It is used to constrain the migration order logic of data blocks within a group. The degree of interaction tightness of the associated edges is used to characterize the comprehensive level of business interaction frequency, time sequence overlap and link coupling strength between two groups of data blocks, and is the core quantitative basis for distinguishing the association priority of data blocks within a group.

[0148] Specifically, a time-series chain construction operation is performed for each data block group. This involves traversing all data block nodes within the group and their corresponding valid association edges, replicating the address dependencies and temporal connections fixed in the data flow graph, eliminating all invalid noise associations and cross-group interference associations, and using the data block that is the first accessed by the business within the group as the head node. Following the order of the original business access to memory, all data block nodes with strong associations are sequentially connected to form a migration time-series chain that is continuous and has a closed temporal loop. This ensures that the topology of the time-series chain is completely consistent with the real multi-core business data flow link. Based on the completion of the time-series chain construction, the interaction tightness of each association edge in the time-series chain is extracted. Using the higher the interaction tightness, the higher the migration priority, as the core sorting rule, all data block nodes in the time-series chain are sorted in order to generate the basic migration order within the group.

[0149] For example, a data block group contains data block A, data block B, and data block C. The interaction strength of the associated edge between data block A and data block B is 0.78, and the interaction strength of the associated edge between data block B and data block C is 0.72. The coupling strength of the other associated edges within the group is even lower. Based on the gradient of interaction strength, the system sorts out the basic migration order as follows: data block A is migrated first, followed by data block B, and finally data block C, forming a stable basic migration time sequence structure within the group.

[0150] By reconstructing the migration timing constraint framework through business associations, a basic migration order is generated based on the degree of quantitative interaction, ensuring that the basic timing conforms to the inherent operating rules of multi-core business and providing a stable and reliable timing benchmark for subsequent dynamic optimization. The basic migration order of each group of data blocks is the timing carrier for subsequent failure urgency optimization and scheduling cost ranking adjustment. The topological rationality of the basic order determines the business adaptability of the migration timing.

[0151] The aforementioned basic migration order is generated solely based on static topological associations and interaction coupling strength, adapting only to steady-state business operation scenarios and not considering the failure decay characteristics and real-time scheduling overhead differences of data blocks themselves. Some short-lifetime data blocks exhibit rapid failure and data iteration characteristics; if the migration is delayed using the basic order, it will lead to temporary data failure and loss, and business data gaps. At the same time, the migration compression scheduling costs of different data blocks vary significantly, and disordered arrangement will result in resource waste and cumulative scheduling overhead. The data failure urgency is a time-sensitive risk level mapped from the data block lifetime, used to characterize the urgency of data block decay and failure destruction over time. The migration position is the sorting position of the data block in the time-series chain within the group. The scheduling cost is used to characterize the system resource consumption level of a single data block migration and storage.

[0152] Specifically, the first layer of sequence optimization is performed. Based on the data block lifecycle, the failure urgency level is divided. Following the aforementioned classification standard, data blocks with a lifecycle of 0 to 0.35 are short-cycle data blocks, corresponding to high failure urgency and a risk of rapid failure; data blocks with a lifecycle of 0.35 to 0.7 are medium-cycle data blocks, corresponding to medium failure urgency and a gradual decay rate; and data blocks with a lifecycle of 0.7 to 1 are long-cycle data blocks, corresponding to low failure urgency and long-term stable validity. Without disrupting the original overall topological association structure of the time-series chain, the basic migration order is locally optimized. Short-cycle data blocks with high failure urgency are appropriately fronted in the time-series chain to prioritize the migration of easily failed data and avoid the risk of data loss. Steady-state data blocks with medium to low urgency retain their original basic order to ensure the continuity of the business chain.

[0153] After completing the failure urgency optimization, the second-level fine-tuning of the ranking is performed. The optimal storage combination scheduling cost corresponding to all data blocks in the group is retrieved. Under the premise of consistent failure urgency level and compatibility of topology association constraints, the ranking is corrected by following the fine-tuning rule of prioritizing migration of low scheduling cost. Under the same time-sensitive risk, data blocks with lower system overhead are prioritized to reduce the computing power and storage resource consumption of the overall batch migration.

[0154] For example, based on the aforementioned basic migration order of A, B, and C, if data block C has a lifespan of 0.28, it belongs to a high-urgency data block and has a risk of rapid failure. While keeping the topological association links of A, B, and C unchanged, the order can be slightly optimized by appropriately advancing the migration position of data block C. If data block B and data block C have the same failure urgency level, and the scheduling cost of data block C is significantly lower than that of data block B, then the position can be finely adjusted within the range allowed by topological constraints, ultimately forming a migration execution order that takes into account topological association, timeliness and safety, and scheduling overhead.

[0155] By optimizing the urgency of failures, the technical risks of short-term data failure and loss are mitigated. By adjusting the scheduling cost priority, the overhead of batch migration is balanced and optimized, achieving a unity of topology adaptability, data security, and scheduling economy, and improving the dynamic adaptive capability of migration timing. The execution order of data block migration serves as the timing execution standard for subsequent batch execution of data migration, compression writing, and multi-layer storage data landing, while also providing timing operation feedback data for the iterative optimization of parameters in the entire scheduling system.

[0156] S4. Migrate the data blocks to the target storage layer according to the migration execution order, and perform compression processing according to the corresponding compression method. Based on the performance indicators obtained after the migration execution, adjust the calculation parameters of the scheduling cost.

[0157] Specifically, adjusting the calculation parameters for scheduling costs includes:

[0158] Data blocks are migrated to the target storage layer according to the migration execution order, and compression processing is performed according to the corresponding compression method;

[0159] The hierarchical storage access latency, multi-core memory access hit rate and compression cost deviation after data block migration and compression are obtained as performance indicators. Based on the performance indicators, the types of deviations such as heat mapping deviation, compression compensation deviation and scheduling operation deviation are identified in reverse.

[0160] Adjust the numerical mapping parameters of the access popularity weight, the compensation parameters of the compression cost, and the joint operation parameters according to the type of deviation.

[0161] During parameter adjustment, the access relationships of data block groups and the criteria for determining their lifecycle are locked.

[0162] The previously output migration sequence, storage level, and compression strategy were all pre-decision results based on historical topology and memory access data, which were only theoretically optimal solutions and could not expose the prediction errors of the algorithm parameters. The migration execution order was the intra-group migration sequence after double-layer optimization of lifecycle urgency and scheduling cost. The target storage level was the matching storage level with the minimum global scheduling cost. The corresponding compression method was the optimal compression algorithm adapted to data characteristics and storage load. Specifically, the topological sequence chain constraint of the locked data block group was used to prevent the disruption of the migration order of related data blocks within the group, thus avoiding the problems of address dependency breakage and data sequence disorder.

[0163] Following the block-by-block sequential migration execution logic, all data blocks are precisely moved to the preset target storage layer. The migration data is compressed and encoded using the matching compression method to ensure data integrity and storage format standardization, completing the cross-layer migration, compression encoding, and layered writing. The migration time, compression computation resource overhead, and final storage location of each data block are recorded to generate a performance log, providing the original data basis for subsequent performance quantification and statistics. For example, for a data block group consisting of strongly correlated data blocks A, B, and C, the migration operation is executed sequentially according to the optimized migration order. Each data block is written to the corresponding high-speed, medium-speed, and low-speed target storage layers, and their respective matching compression algorithms are executed to form a stable storage layout.

[0164] The above steps complete the accurate transformation from theoretical scheduling strategy to actual operating conditions, generate real and statistically verifiable hierarchical storage operation samples, provide a source of data for subsequent performance index quantification, deviation identification and parameter correction, and the standardization of implementation determines the accuracy of iterative optimization.

[0165] Theoretical scheduling parameters are generated based on historical data statistics, and there will inevitably be systematic errors compared with real-time dynamic operating conditions; tiered storage access latency is the average response time of multi-core processor cores accessing and writing data blocks, normalized to the range of 0 to 1; multi-core memory access hit rate is the normalized ratio of effective memory accesses to total memory accesses; compression cost deviation is the normalized difference between the theoretically calculated compression cost and the actual compression cost; for example, if 0.12 is set as the deviation judgment threshold, if the absolute value of the performance error is greater than 0.12, it is judged as having a valid systematic deviation, and if it is less than or equal to 0.12, it is judged as normal operating condition fluctuation, and no parameter adjustment is triggered.

[0166] Specifically, the tiered storage access latency deviation is calculated as: |Normalized Actual Access Latency - Normalized Theoretical Access Latency|; the multi-core memory access hit rate deviation is calculated as: |Normalized Actual Hit Rate - Normalized Theoretical Hit Rate|; and the compression cost deviation is calculated as: |Normalized Actual Compression Cost - Normalized Theoretical Compression Cost|. A corresponding deviation tracing and judgment mechanism is established to achieve accurate classification and identification. When the access latency deviation > 0.12 and the hit rate deviation is within the normal fluctuation range, a hotness mapping deviation is identified. This deviation stems from an unreasonable ratio between the sharing degree, lifecycle interval mapping parameters, and access demand correction coefficient, leading to incorrect data hotness / coldness classification and storage layer mismatch. For example, if the normalized theoretical access latency of a core data block is 0.21 and the actual access latency is 0.35, the deviation is 0.14, which is greater than 0.12. Other indicators are normal, and this is accurately identified as a hotness mapping deviation.

[0167] When the hit rate deviation is greater than 0.12 and the compression cost deviation is normal, it is determined that there is a compression compensation deviation. This deviation is caused by the mismatch of the dynamic compensation coefficient of the storage layer load. Insufficient compensation in high load scenarios and excessive compensation in low load scenarios lead to a decrease in compression adaptability. For example, the theoretical memory access hit rate of a certain data block is 0.83, the actual hit rate is 0.69, the deviation value is 0.14 > 0.12, and the system identifies it as a compression compensation deviation.

[0168] When the compression cost deviation is greater than 0.12, and there is no structural deviation in latency and hit rate, it is determined that there is a scheduling operation deviation. This deviation is caused by the imbalance in the joint operation weight ratio of the migration data volume and the compression cost, resulting in the distortion of the comprehensive scheduling cost calculation. For example, the theoretical average compression cost of a certain data block group is 0.36, the actual average compression cost is 0.5, and the deviation value of 0.14 > 0.12. The system identifies this as a scheduling operation deviation.

[0169] A multi-cycle continuous sampling verification mechanism is adopted. Only when the deviation threshold is triggered in three consecutive 10ms sampling cycles is it determined to be a valid deviation, eliminating invalid errors caused by instantaneous hardware fluctuations and sudden task interference, thus ensuring the rigor of deviation identification. The above steps construct a quantifiable, traceable, and categorizable deviation identification system, providing an accurate classification basis for targeted parameter correction. Deviation type is the core prerequisite for subsequent fine-tuning of directional parameters, determining the targeting and effectiveness of iterative optimization.

[0170] The three types of deviations correspond to independent computing units in the scheduling system. In this embodiment, the single parameter fine-tuning amplitude is set to no more than 0.05 to avoid scheduling system oscillations caused by parameter mutations. All fine-tuning continues the 0~1 system. For the heat mapping deviation, the numerical mapping parameters of the access heat weight are adjusted, specifically including the interval level coefficients of sharing degree and life cycle and the access demand correction coefficient. When a positive latency deviation occurs, it indicates that high-value data is misallocated to the low-speed storage layer. The high interval level coefficient is slightly increased and the low interval level coefficient is slightly decreased, and the correction coefficient ratio of high-intensity access demand is optimized simultaneously. For example, if the original high interval level coefficient is 0.9 and the low interval is 0.3, after a single fine-tuning, the high interval coefficient is adjusted to 0.93 and the low interval coefficient is adjusted to 0.28, correcting the heat mapping deviation and optimizing the accuracy of hot and cold data classification.

[0171] For compression compensation deviation, adjust the dynamic compensation parameters of the storage layer load, where the load compensation coefficient = 1 + real-time load rate × 0.3. If insufficient compensation persists in high-load scenarios, slightly increase the multiplier constant by 0.3; if overcompensation occurs in low-load scenarios, slightly decrease the multiplier constant. For example, for high-load compensation deviation, fine-tune the fixed multiplier from 0.3 to 0.34 to enhance the compression cost compensation under high-load conditions and adapt to the actual resource consumption. For scheduling operation deviation, adjust the joint operation weighting parameters of migration data volume and compression cost. The original equal weighting ratio was 0.5, 0.5. If the theoretical cost is consistently low, it indicates that the compression cost weight is insufficient, so slightly increase the compression cost weight and decrease the migration data volume weight. If the theoretical cost is consistently high, make a reverse fine-tuning. For example, when there is a positive scheduling operation deviation, adjust the weighting ratio to 0.47 for migration data volume and 0.53 for compression cost to correct the joint operation distortion problem and narrow the difference between theoretical and actual costs.

[0172] The above steps continuously converge errors while retaining the original architecture, steadily improving the overall accuracy of multi-layer storage matching, compression cost calculation, and scheduling decisions. The updated calculation parameters will serve as the standard for all data scheduling operations in the next cycle, determining the accuracy of subsequent full-process scheduling schemes.

[0173] The entire scheduling system consists of two types of parameters: dynamic iterative parameters and static inherent benchmarks. Dynamic parameters can be adaptively fine-tuned according to operating conditions, while data topology association and timeliness judgment benchmarks are inherent attributes of the data. Once changed, they will lead to the confusion of the global statistical caliber, the failure of the topology structure, and the inconsistency of the algorithms before and after. Specifically, throughout the entire cycle of all dynamic parameter fine-tuning iterations, the effective access association edges, the degree of interaction, and the clustering and aggregation results of the global data blocks are locked, and any changes to the topology structure, association strength, and grouping relationship are prohibited. The boundaries of the three equal intervals of the life cycle, the corresponding level coefficients of the intervals, and the joint operation judgment formula are locked simultaneously, and the static judgment criteria are not adjusted with performance deviations or changes in operating conditions.

[0174] For example, regardless of the parameter fine-tuning performed, the interval boundaries of the lifecycle (0~0.35 short cycle, 0.35~0.7 medium cycle, 0.7~1 long cycle) are permanently fixed, the chain-like topological association structure of the data block group remains unchanged, and only iterable parameters such as heat mapping coefficient, compensation multiplier and operation weight are dynamically adjusted.

[0175] The above steps construct a hierarchical optimization system with constant static benchmarks and iterative dynamic parameters, ensuring that each iteration of optimization is based on a unified, stable, and reliable technical foundation. The static benchmark system is the core foundation for the long-term stable operation and continuous iterative optimization of the entire scheduling system, ensuring the uniformity and standardization of scheduling throughout the entire cycle.

[0176] Example 2

[0177] like Figure 3 As shown, this embodiment provides a multi-layer storage architecture data scheduling system based on a multi-core processor, including an acquisition module, an analysis module, a scheduling module, and an execution module.

[0178] The acquisition module is used to acquire the memory access event stream of each processor core, identify the execution phase changes of the processor core, predict the access requirements of the corresponding processor core, and acquire the operating status of each storage layer in the storage system.

[0179] The analysis module is used to perform address dependency and timing correlation analysis on all memory access event streams, construct a data flow graph with data blocks as nodes, and determine the sharing degree, affinity and lifespan of data blocks based on the topological characteristics of the data flow graph and the inter-core access distribution and time span of different data blocks.

[0180] The scheduling module is used to calculate the scheduling cost and determine the target storage layer and corresponding compression method based on the sharing degree, affinity, life cycle and access requirements of data blocks, combined with the operating status of the storage layer and the amount of data to be migrated and the compression cost under the compression method. It also determines the migration execution order based on the access relationship of data blocks within the data block group.

[0181] The execution module is used to migrate data blocks to the target storage layer according to the migration execution order, perform compression processing according to the corresponding compression method, and adjust the calculation parameters of the scheduling cost based on the performance indicators obtained after the migration execution.

[0182] The above description is merely a preferred embodiment of this application. The scope of protection of this application is not limited to the above embodiments. All technical solutions falling within the scope of this application's concept are within the scope of protection of this application. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of this application should also be considered within the scope of protection of this application.

Claims

1. A data scheduling method for a multi-layered storage architecture based on a multi-core processor, characterized in that, include: The memory access event stream of each processor core is obtained, the execution phase changes of the processor core are identified, the access requirements of the corresponding processor core are predicted, and the operating status of each storage layer in the storage system is obtained. Address dependency and timing correlation analysis are performed on all memory access event streams. A data flow graph with data blocks as nodes is constructed. Based on the topological characteristics of the data flow graph and the inter-core access distribution and effective access duration of different data blocks, the sharing degree, affinity and lifespan of data blocks are determined. Based on the sharing degree, affinity, lifespan and access requirements of data blocks, combined with the operating status of the storage layer and the amount of data to be migrated and the compression cost under the compression method, the scheduling cost is calculated and the target storage layer and corresponding compression method are determined. The migration execution order is determined according to the access relationship of data blocks within the data block group. Data blocks are migrated to the target storage layer according to the migration execution order, and compression processing is performed according to the corresponding compression method. Based on the performance indicators obtained after the migration execution, the calculation parameters of the scheduling cost are adjusted.

2. The data scheduling method for a multi-layered storage system based on a multi-core processor according to claim 1, characterized in that, The predicted access requirements for the corresponding processor cores include: The memory access events of the processor core are acquired and concatenated to form a memory access event stream, and the hardware operation data of the processor core within the same sampling period are acquired. By comparing the distribution characteristics of memory access events with the numerical changes of hardware operation data within adjacent sampling periods, the execution phase changes of the processor core are identified based on the comparison results. Based on the execution phase change, the behavioral characteristics of memory access events under the current execution phase are extracted, and the feature weights are set according to the phase duration to obtain the access status of the processor core. The system analyzes data sharing and resource contention behaviors among processor cores, adjusts access patterns based on the statistical results, and predicts the access demands of corresponding processor cores.

3. The data scheduling method for a multi-layered storage system based on a multi-core processor according to claim 1, characterized in that, Constructing the data flow graph includes: It aggregates the memory access event streams from all processor cores and groups the memory access events to the corresponding data blocks according to their physical addresses; Analyze the address dependencies and timing relationships of cross-core memory access, filter and retain the access relationships between data blocks to form the association edges between data blocks; Using data blocks as nodes, node connections are established based on the parsed address dependencies and temporal sequence relationships, and feature attributes are labeled for the associated edges according to the degree of interaction between the associated edges. Integrate the nodes, associated edges, and characteristic attributes of all data blocks to form a data flow graph.

4. The data scheduling method for a multi-layered storage system based on a multi-core processor according to claim 3, characterized in that, The determination of data block sharing degree, affinity, and lifespan includes: Extract the topological features of nodes from the data flow graph, and statistically analyze the inter-core access distribution of data blocks and the effective access duration within the sampling period; The sharing degree of a data block is determined by combining the node coverage area and the effective status of associated edges with the number of processor cores accessing the corresponding data block and the effective access duration. Based on the valid state of the associated edges, and combined with the access ratio and valid access duration of a single processor core to the data block, the affinity between the processor core and the data block is obtained by comprehensive comparison. The system determines the existence status of nodes, combines the distribution activity of data block access with the effective access duration for joint calculation, and determines the life cycle of data blocks based on the calculation results.

5. The data scheduling method for a multi-layered storage system based on a multi-core processor according to claim 4, characterized in that, The topological features include node coverage, existence status, and the valid status of associated edges; the inter-core access distribution includes the number of processor cores accessing data blocks, the proportion of data blocks accessed by a single processor core, and the distribution activity of data block access. The distribution activity is obtained based on the access frequency of the processor cores. Based on the access relationships between data blocks, the data blocks are grouped together to determine the data block group.

6. The data scheduling method for a multi-layered storage system based on a multi-core processor according to claim 5, characterized in that, The determination of the target storage layer and corresponding compression method includes: Using the sharing degree and lifespan of data blocks as constraints, the access popularity weight of data blocks is determined in combination with access requirements, and the operating status of each storage layer in the storage system is matched to obtain the initial matching result. The initial matching results are corrected based on the affinity between the processor core and the data block to obtain candidate storage layers. The amount of migrated data and the compression cost under different compression methods are traversed, and the compression cost is dynamically compensated based on the current load status of the candidate storage layers to obtain the scheduling cost. By comparing the scheduling costs of each candidate storage layer with different compression methods, the storage combination with the lowest scheduling cost is selected as the target storage layer and corresponding compression method.

7. The data scheduling method for a multi-layered storage architecture based on a multi-core processor according to claim 6, characterized in that, The access popularity weight is generated by defining the value range of data block sharing degree and life cycle respectively, using the value range as the constraint boundary, and combining access requirements to generate a numerical mapping. The scheduling cost is generated by jointly calculating the amount of migrated data under the current compression method and the compensated compression cost; the storage combination includes the storage layer and its corresponding compression method.

8. The data scheduling method for a multi-layered storage system based on a multi-core processor according to claim 7, characterized in that, Determining the migration execution order includes: The migration sequence chain of each group of data blocks is constructed based on the access relationship, and the basic migration order of data blocks within the data block group is determined by the degree of interaction of the associated edges. The urgency of data failure is determined by quantifying the lifecycle of data blocks to optimize the basic migration order. The migration order is also adjusted by considering the scheduling cost of data blocks to determine the migration execution order.

9. The data scheduling method for a multi-layered storage system based on a multi-core processor according to claim 8, characterized in that, The calculation parameters for the adjusted scheduling cost include: The hierarchical storage access latency, multi-core memory access hit rate and compression cost deviation after data block migration and compression are obtained as performance indicators. Based on the performance indicators, the types of deviations such as heat mapping deviation, compression compensation deviation and scheduling operation deviation are identified in reverse. Adjust the numerical mapping parameters of the access popularity weight, the compensation parameters of the compression cost, and the joint operation parameters according to the type of deviation. During parameter adjustment, the access relationships of data block groups and the criteria for determining their lifecycle are locked.

10. A multi-layered storage architecture data scheduling system based on a multi-core processor, used to implement the multi-layered storage architecture data scheduling method based on a multi-core processor as described in any one of claims 1-9, characterized in that, include: The module comprises an acquisition module, an analysis module, a scheduling module, and an execution module. The acquisition module is used to acquire the memory access event stream of each processor core, identify the execution phase changes of the processor core, predict the access requirements of the corresponding processor core, and acquire the operating status of each storage layer in the storage system. The analysis module is used to perform address dependency and timing correlation analysis on all memory access event streams, construct a data flow graph with data blocks as nodes, determine the sharing degree, affinity and lifespan of data blocks based on the topological characteristics of the data flow graph and the inter-core access distribution and time span of different data blocks, and determine data block groups based on access correlations; The scheduling module is used to calculate the scheduling cost and determine the target storage layer and corresponding compression method based on the sharing degree, affinity, life cycle and access requirements of data blocks, combined with the operating status of the storage layer and the amount of data to be migrated and the compression cost under the compression method. It also determines the migration execution order based on the access relationship of data blocks within the data block group. The execution module is used to migrate data blocks to the target storage layer according to the migration execution order, perform compression processing according to the corresponding compression method, and adjust the calculation parameters of the scheduling cost based on the performance indicators obtained after the migration execution.

Images