Information processing method and system applied to all-flash file storage

Abstract

The present invention provides an information processing method and system for all-flash storage of files. The method first obtains storage request information received by a target storage node, which includes a set of data blocks to be processed and a set of expected storage attributes, thereby describing the configuration requirements of storage capacity distribution, access path optimization, and data compression level. Then, multi-dimensional features of the set of data blocks to be processed are extracted to generate a data block feature vector, and the attributes of the set of expected storage attributes are encoded to generate a storage attribute feature vector. The two are input into a pre-trained storage strategy generation model for dynamic weight fusion, and a target storage strategy parameter set that meets the configuration requirement constraints is output. Finally, relevant parameters are generated based on the information, and the all-flash storage system is driven to perform distributed storage operations on the data block set, thereby realizing efficient and intelligent all-flash storage information processing.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to an information processing method and system applied to all-flash file storage. Background Art

[0002] With the rapid development of information technology, data volumes are exploding, placing higher demands on the performance, efficiency, and flexibility of storage systems. As a new storage technology, all-flash storage, with its high-speed data read and write capabilities, is gaining widespread adoption in various data storage scenarios.

[0003] Traditional file storage technology handles storage requests in a simplistic and single-minded manner. Storage resources are typically allocated solely based on the size of the data, lacking detailed management of data distribution across storage capacity. For example, data blocks are simply stored in a sequential or fixed storage node allocation pattern, without fully considering the potential differences in access frequencies of different data blocks and the load balancing of different storage nodes. This results in some storage nodes being overloaded while others are idle, resulting in inefficient overall storage resource utilization.

[0004] When it comes to access path optimization, existing technologies mostly use static path planning. Once the data storage location is determined, the access path is essentially fixed and cannot be dynamically adjusted based on real-time system load and data access patterns. This can lead to increased data access latency and slower response speeds in complex and ever-changing business scenarios, such as when large numbers of users simultaneously access product data during e-commerce promotions or when popular series updates on video websites trigger high concurrent access. This can severely impact the user experience. Summary of the Invention

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides an information processing method applied to all-flash file storage, the method comprising:

[0006] Obtaining storage request information received by a target storage node, the storage request information including a set of data blocks to be processed and a corresponding set of desired storage attributes; wherein the set of desired storage attributes is used to describe configuration requirements of the set of data blocks to be processed in terms of storage capacity distribution, access path optimization, and data compression level;

[0007] Performing multi-dimensional feature extraction on the set of data blocks to be processed to generate a data block feature vector, and performing attribute encoding on the set of expected storage attributes to generate a storage attribute feature vector;

[0008] Inputting the data block feature vector and the storage attribute feature vector into a pre-trained storage policy generation model, performing dynamic weight fusion on the data block feature vector and the storage attribute feature vector through the storage policy generation model, and outputting a target storage policy parameter set;

[0009] Generate storage node allocation instructions, path priority configuration and compression execution parameters according to the target storage policy parameter set, and drive the all-flash storage system to perform distributed storage operations on the set of data blocks to be processed based on the storage node allocation instructions; wherein, the target storage policy parameter set meets the constraints of all configuration requirements in the expected storage attribute set.

[0010] On the other hand, an embodiment of the present invention also provides an information processing system for all-flash file storage, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0011] Based on the above aspects, the embodiment of the present application realizes the intelligent generation of storage policies and the efficient execution of storage operations. Specifically, the storage request information received by the target storage node is first obtained, which includes the set of data blocks to be processed and the corresponding expected storage attribute set. The expected storage attribute set clearly describes the configuration requirements of the set of data blocks to be processed in terms of storage capacity distribution, access path optimization and data compression level. On this basis, multi-dimensional feature extraction is performed on the set of data blocks to be processed to generate a data block feature vector. At the same time, attribute encoding is performed on the expected storage attribute set to generate a storage attribute feature vector. The data block feature vector and the storage attribute feature vector are input into the pre-trained storage policy generation model. The storage policy generation model analyzes the relationship between the two through a dynamic weight fusion mechanism and outputs a target storage policy parameter set. It can flexibly adjust the influence of each feature in the policy generation according to the characteristics of different data blocks and diversified storage requirements, thereby generating a more accurate and effective storage policy parameter set. Then, storage node allocation instructions, path priority configuration and compression execution parameters are generated according to the target storage policy parameter set, and the all-flash storage system is driven to perform distributed storage operations on the set of data blocks to be processed based on the storage node allocation instructions. Because the target storage policy parameter set satisfies the constraints of all configuration requirements in the desired storage attribute set, the entire storage operation can be performed precisely as the user expects. Therefore, through the intelligent storage policy generation model, data characteristics and storage requirements can be automatically and accurately matched, avoiding the tediousness and errors of manual policy configuration, and greatly improving the efficiency and accuracy of storage management. At the same time, the dynamic weight fusion mechanism makes the storage policy highly flexible and adaptable, able to cope with complex storage requirements in different scenarios. During the distributed storage operation execution phase, based on the precise storage policy parameter set, storage nodes can be reasonably allocated, access paths can be optimized, and appropriate compression parameters can be set, thereby fully leveraging the performance advantages of the all-flash storage system, improving the efficiency of data storage and access, and reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The figure is a schematic diagram of the execution flow of an information processing method applied to all-flash file storage provided by an embodiment of the present invention.

[0013] Figure 2 FIG. 4 is a schematic diagram of exemplary hardware and software components of an information processing system for all-flash file storage provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 FIG1 is a flow chart of an information processing method for all-flash file storage provided by an embodiment of the present invention. The information processing method for all-flash file storage is introduced in detail below.

[0015] Step S110: Obtain storage request information received by the target storage node, wherein the storage request information includes a set of data blocks to be processed and a corresponding set of desired storage attributes. The desired storage attribute set is used to describe the configuration requirements of the set of data blocks to be processed in terms of storage capacity distribution, access path optimization, and data compression level.

[0016] In this embodiment, a data center of an Internet company uses an all-flash storage system to store massive data, and a target storage node receives storage request information from different business departments within the Internet company.

[0017] For example, a company's video business department has a large amount of video material that needs to be stored. This video material constitutes the set of data blocks to be processed. These include video files of different resolutions (such as HD and SD), different lengths (from short videos of a few seconds to long videos of several hours), and different formats (such as MP4, AVI, etc.). Each video file is considered a data block. Based on this, the corresponding set of desired storage attributes is considered. In terms of storage capacity distribution, due to the huge amount of video data, it is hoped that it can be evenly distributed across various storage nodes to avoid excessive storage pressure on any one node. For example, it is expected that the total video data will be distributed to different storage nodes according to a certain ratio (such as evenly distributed to three storage nodes, with each node storing approximately one-third of the data volume).

[0018] In terms of access path optimization, for popular videos (determined based on historical playback statistics, such as recently popular short videos), a shorter access path is desired to quickly respond to client playback requests. For less frequently accessed video material, the access path can be relatively long, but it must also ensure that it can be retrieved within a certain response time.

[0019] In terms of data compression level, for some high-resolution, high-quality videos, since they have already been compressed to a certain extent and need to maintain a high image quality, it is expected to use a lower compression level, such as a compression ratio of 2:1; while for some low-resolution videos with low image quality requirements, a higher compression level, such as a compression ratio of 10:1, can be used to save storage space.

[0020] For example, a company's finance department needs to store financial statement data. This report data, a collection of data blocks to be processed, includes monthly, quarterly, and annual financial statements, which may be formatted as Excel spreadsheets or PDF files. In terms of storage capacity distribution, due to the importance and relevance of financial statement data, it may be desirable to store reports from the same year on similar storage nodes for ease of management and querying. Regarding access path optimization, finance department personnel may frequently need to query reports from the most recent quarters, so the access paths to these reports must be optimized to ensure fast access. Regarding data compression levels, financial statement data must maintain accuracy, so a lower compression level, such as a 3:1 ratio, may be used to avoid data distortion.

[0021] Step S120 , performing multi-dimensional feature extraction on the set of data blocks to be processed to generate a data block feature vector, and performing attribute encoding on the set of expected storage attributes to generate a storage attribute feature vector.

[0022] Specifically, for the video data block collection of the video business department, when generating the data block feature vector through multidimensional feature extraction, the content distribution pattern of each video data block is analyzed, such as the color distribution of the video image (is it a colorful movie image or a relatively monochromatic documentary image) and the degree of dynamics (action films tend to be more dynamic, while landscapes tend to be more static). Metadata features are also statistically analyzed, such as data block size distribution (HD videos are significantly larger than SD videos); hot and cold access tags (using play volume to identify popular videos as hot data blocks and infrequently played videos as cold data blocks); and associated file types, such as subtitle files and cover image files, are also analyzed. Based on these metadata features, an initial feature set is constructed, and then temporal correlation analysis is performed. For example, if it is found that the views of specific video types (such as Lunar New Year films) increase significantly during certain holidays, the dynamic weight coefficients of the relevant feature dimensions of these videos will be increased during that time period. The initial feature set is denoised based on a dynamic weight coefficient. For example, if the weight of a metadata feature that is not closely related to the video content is lower than a preset threshold, such as the associated tag weight of a rarely used video format, it will be removed to generate an optimized data block feature vector.

[0023] For the desired storage attribute set, if the storage capacity distribution sub-attribute is a numeric metric, assuming it is expressed as a percentage of the available capacity of the storage node, it is mapped to a pre-defined dimensional feature space through piecewise normalization. For example, mapping 0-20% available capacity to one feature value, 20-40% to another, and so on. For the logical metric of access path optimization, if access paths are categorized as fast, medium, and slow, binary vectorization is used to generate a discrete feature sequence, such as 100 for fast, 010 for medium, and 001 for slow. For the data persistence level sub-attribute, if it is categorized as high, medium, and low, a similar binary vectorization is performed. Then, weighted concatenation is performed based on metric priority (e.g., storage capacity distribution has the highest priority, followed by data persistence level, and finally access path optimization) to generate a global storage attribute feature vector.

[0024] For the finance department's financial report data block collection, the content distribution pattern of each report data block is analyzed, such as the distribution of data types within the report (whether it primarily contains income and expenditure data or assets and liabilities data, etc.). Metadata features are statistically analyzed, such as data block size distribution (annual report data blocks are larger, monthly reports are relatively smaller), hot and cold access tags (recent reports are hot data blocks, and those from years ago are cold data blocks), and associated file types (such as report annotation files). After constructing the initial feature set, a time series correlation analysis was performed, revealing that the frequency of access to relevant reports increases during the end-of-quarter and year-end financial settlement periods, and the dynamic weight coefficients of the corresponding feature dimensions increase. Noise reduction processing is performed to remove unimportant metadata features. For example, a specific formatting tag in a rarely viewed report has a low weight. This tag is removed to generate a data block feature vector.

[0025] Numerical indicators of storage capacity distribution within the desired storage attribute set (measured by the number of bytes of free space on storage nodes) are segmented and normalized and mapped to a pre-defined dimensional feature space. Logical indicators for access path optimization (e.g., categorized into three levels: emergency access, daily access, and infrequent access) are binary vectorized. Data persistence levels (e.g., required long-term storage, short-term storage, and deletable as needed) are also binary vectorized and then weighted by priority to generate a storage attribute feature vector.

[0026] Step S130: input the data block feature vector and the storage attribute feature vector into a pre-trained storage policy generation model, dynamically weight the data block feature vector and the storage attribute feature vector through the storage policy generation model, and output a target storage policy parameter set.

[0027] In this embodiment, continuing with the video business department as an example, the feature vectors of video data blocks and storage attribute feature vectors are input into a pre-trained storage policy generation model. The feature interaction layer within the model processes the two vectors interactively, for example, identifying the relationship between features such as video data block size and hot / cold access flags and storage attributes such as storage capacity distribution and access path optimization. The policy prediction layer predicts storage policy parameters based on these relationships. For example, popular videos with large data blocks may be assigned to storage nodes with larger storage capacity and faster access speeds. Path priority configuration may be given a higher priority to ensure fast access. Compression execution parameters may be set based on the previously determined lower compression level (e.g., 2:1). Cold videos with smaller data blocks may be assigned to storage nodes with relatively smaller storage capacity and slower access speeds, with lower path priority and higher compression execution parameters (e.g., 10:1). The constraint verification layer verifies whether the predicted policy parameters meet the configuration requirements of the desired storage attribute set. For example, it verifies whether the allocated storage nodes meet storage capacity distribution requirements, whether the latency corresponding to the predicted path priority meets the latency threshold requirements in access path optimization, and whether the predicted compression execution parameters meet the data compression level requirements. If not, the model adjusts the weights and re-performs dynamic weight fusion until the output is a target storage policy parameter set that meets all configuration requirements.

[0028] For the finance department's financial report data, after inputting it into the model, the model identifies the relationship between financial report data block characteristics (such as block size and hot / cold access identifiers) and storage attributes (such as storage capacity distribution, preferably storing annual reports together, and optimizing access paths for recent reports). The policy prediction layer predicts storage policy parameters. For recent, important financial reports (such as annual reports), storage nodes with high reliability and fast access speeds are allocated, path priority is set to the highest, and compression parameters are set to a lower compression level (such as 3:1). For less important reports from years ago, storage node allocation can be more flexible, path priority is lower, and compression parameters can be used to increase the compression ratio while ensuring data accuracy. The constraint verification layer also verifies whether the predicted policy parameters meet the configuration requirements of the desired storage attribute set. For example, whether the storage node allocation meets the storage capacity distribution requirements, whether the access speed corresponding to the path priority meets the finance department's needs, and whether the compression parameters ensure data accuracy. By continuously adjusting the weights and integrating them, the target storage policy parameter set is output.

[0029] Step S140: Generate a storage node allocation instruction, path priority configuration, and compression execution parameters based on the target storage policy parameter set, and drive the all-flash storage system to perform a distributed storage operation on the set of data blocks to be processed based on the storage node allocation instruction. The target storage policy parameter set satisfies the constraints of all configuration requirements in the desired storage attribute set.

[0030] Continuing with the video business department as an example, storage node allocation instructions are generated based on the target storage policy parameter set. For popular and large video data blocks, the all-flash storage system allocates these blocks to storage nodes with large capacity and high read and write speeds based on the storage node allocation parameters. Regarding path priority configuration, the access path for these popular videos is set to high priority. In the all-flash storage system's network topology, the path from the client to the storage node is ensured to be the shortest or to traverse the fewest network devices. Regarding compression execution parameters, video data blocks are compressed and stored at a set low compression level (e.g., 2:1). This allows the all-flash storage system to store video data as intended during distributed storage operations, meeting both storage capacity distribution requirements (avoiding excessive storage pressure on any particular node), access path optimization (fast access to popular videos), and data compression level requirements (low compression levels for videos requiring high image quality).

[0031] For the finance department's financial report data, storage node allocation instructions are generated based on the target storage policy parameter set, assigning recent important reports (such as annual reports) to dedicated storage nodes with high reliability and fast access. In path priority configuration, the access path for recent reports is set to the highest priority to ensure that finance personnel can quickly query them. Compression execution parameters compress report data at a low compression level (such as 3:1) before storage. When the all-flash storage system performs distributed storage operations, it stores the financial report data according to this storage node allocation instruction, meeting all requirements for storage capacity distribution (storing together with annual reports), access path optimization (fast access to recent reports), and data compression level (ensuring data accuracy).

[0032] Based on the above steps, the embodiment of the present application realizes the intelligent generation of storage policies and the efficient execution of storage operations. Specifically, first, the storage request information received by the target storage node is obtained, which includes the set of data blocks to be processed and the corresponding expected storage attribute set. The expected storage attribute set clearly describes the configuration requirements of the set of data blocks to be processed in terms of storage capacity distribution, access path optimization and data compression level. On this basis, multi-dimensional feature extraction is performed on the set of data blocks to be processed to generate a data block feature vector. At the same time, attribute encoding is performed on the expected storage attribute set to generate a storage attribute feature vector. The data block feature vector and the storage attribute feature vector are input into the pre-trained storage policy generation model. The storage policy generation model analyzes the relationship between the two through a dynamic weight fusion mechanism and outputs a target storage policy parameter set. It can flexibly adjust the influence of each feature in the policy generation according to the characteristics of different data blocks and diverse storage requirements, thereby generating a more accurate and effective storage policy parameter set. Then, storage node allocation instructions, path priority configuration and compression execution parameters are generated according to the target storage policy parameter set, and the all-flash storage system is driven to perform distributed storage operations on the set of data blocks to be processed based on the storage node allocation instructions. Because the target storage policy parameter set satisfies the constraints of all configuration requirements in the desired storage attribute set, the entire storage operation can be performed precisely as the user expects. Therefore, through the intelligent storage policy generation model, data characteristics and storage requirements can be automatically and accurately matched, avoiding the tediousness and errors of manual policy configuration, and greatly improving the efficiency and accuracy of storage management. At the same time, the dynamic weight fusion mechanism makes the storage policy highly flexible and adaptable, able to cope with complex storage requirements in different scenarios. During the distributed storage operation execution phase, based on the precise storage policy parameter set, storage nodes can be reasonably allocated, access paths can be optimized, and appropriate compression parameters can be set, thereby fully leveraging the performance advantages of the all-flash storage system, improving the efficiency of data storage and access, and reducing storage costs.

[0033] In a possible implementation, the desired storage attribute set includes at least one sub-attribute group, and each sub-attribute group corresponds to a storage performance indicator type. Step S120 includes:

[0034] Step S121 : For each sub-attribute group, a corresponding encoding rule is selected according to its corresponding storage performance indicator type.

[0035] Step S122: If the sub-attribute group is a numerical indicator, it is mapped to a preset dimensional feature space through piecewise normalization. If the sub-attribute group is a logical indicator, a discrete feature sequence is generated through binary vectorization.

[0036] Step S123: weighting and concatenating the encoded feature sequences of each sub-attribute group according to the indicator priority to generate a global storage attribute feature vector. The storage performance indicator types include access delay threshold, data persistence level, and storage node load balancing coefficient.

[0037] Taking the video data storage scenario of the video business department as an example, the expected storage attribute set contains multiple sub-attribute groups, each of which corresponds to a storage performance indicator type.

[0038] Assume that an all-flash storage system has three storage nodes, labeled Node A, Node B, and Node C. The storage node load balancing factor is measured by the ratio of each node's currently used storage capacity to the total storage capacity. Node A has 30% of its capacity used, Node B has 25%, and Node C has 20%. Assuming the total storage capacity is 1000 GB (for illustrative purposes only), Node A has 300 GB used, Node B has 250 GB used, and Node C has 200 GB used.

[0039] You can set segmented normalization rules, for example, mapping 0-30% of used capacity to eigenvalue 1, 30%-60% to eigenvalue 2, and 60%-100% to eigenvalue 3. The eigenvalue corresponding to the load balancing coefficient of node A is 1, and the eigenvalues ​​for nodes B and C are also 1.

[0040] Data persistence levels are categorized as high, medium, and low. High means data must be stored long-term with redundant backups, medium means data must be stored for a limited period with a limited number of backups, and low means data can be deleted or transferred at any time based on storage resource availability. If the video business department requires high-level data persistence for popular video material, it will be binary-vectorized, with the high level represented by 110 (the binary encoding here is custom, with 110 representing a high-level encoding method. The first digit may indicate whether there are redundant backups, the second digit indicates the length of the retention period, and the third digit indicates whether it can be adjusted based on resource availability).

[0041] For access latency thresholds, assume they are categorized into three levels: fast, medium, and slow. Fast means that when a client requests video playback, data transmission must begin within 1 second, medium within 3 seconds, and slow within 5 seconds. For popular videos, a fast access latency threshold is desired, and its binary vectorization is set to 100 (again, using custom encoding; the first digit indicates whether the 1-second speed requirement is met; the second and third digits can represent other relevant latency attributes).

[0042] Among them, the indicator priority can be set. Assume that the storage node load balancing coefficient priority is 3, the data persistence level priority is 2, and the access delay threshold priority is 1.

[0043] For the load balancing coefficient characteristic value of 1, since the priority is 3, the calculated weighted value is 1×3 = 3. For the binary vectorization result of the data persistence level 110, it is converted to a decimal number (calculation process: 1×2²+1×2¹+0×2 0 = 6), multiplied by the priority 2 to get 12. For the binary vectorization result of the access latency threshold 100, convert it to decimal (calculation process: 1×2²+0×2¹+0×2 0 = 4), multiplied by the priority 1 to get 4.

[0044] Next, these weighted results are concatenated sequentially (for example, by storage node load balancing coefficient, data persistence level, and access latency threshold) to obtain the storage attribute feature vector. Assume the concatenated result is 3 - 12 - 4 (the "-" here is used simply to indicate the concatenation order).

[0045] For a finance department's financial report data storage scenario, assume that an all-flash storage system has three storage nodes for storing financial report data. Node A stores 40% of the financial report data (assuming the total data volume is 100 GB and Node A stores 40 GB), Node B stores 30% (30 GB), and Node C stores 30% (30 GB). A segmented normalization rule is set, with eigenvalue 1 for the range 0-35% and eigenvalue 2 for the range 35%-70%. The eigenvalue corresponding to the load balancing coefficient for Node A is 2, while that for Nodes B and C is 1.

[0046] Assume that data persistence is categorized into three levels: long-term retention required, short-term retention, and optional deletion. The finance department's annual report requires long-term retention, so its binary vectorization value is 100 (a custom encoding where the first digit indicates whether long-term retention is required, and the second and third digits indicate other relevant attributes).

[0047] For access latency thresholds, we assume three levels: urgent access (response within 1 second), daily access (response within 3 seconds), and rare access (response within 5 seconds). For recent financial statements, the latency threshold for urgent access is expected, and its value is binary vectorized to 100.

[0048] You can set the storage node load balancing coefficient priority to 3, the data persistence level priority to 2, and the access delay threshold priority to 1.

[0049] For the load balancing coefficient, the eigenvalue of node A is 2 multiplied by the priority 3 to get 6, and the eigenvalues ​​of nodes B and C are 1 multiplied by the priority 3 to get 3. For the data persistence level, the binary vectorization result 100 is converted to a decimal number (calculation process: 1×2²+0×2¹+0×2 0 = 4), multiplied by the priority 2 to get 8. The binary vectorization result 100 of the access delay threshold is converted to a decimal number (calculation process: 1×2²+0×2¹+0×2 0 = 4), multiplied by the priority 1 to get 4.

[0050] Thus, the storage attribute feature vector is obtained by sequentially splicing, which is assumed to be 6 - 8 - 4 (the weighted value of the load balancing coefficient of node A - the weighted value of the data persistence level - the weighted value of the access delay threshold).

[0051] Step S124 , analyzing the content distribution pattern of each data block in the set of data blocks to be processed, and counting metadata features of each data block, wherein the metadata features include data block size distribution, hot and cold access marks, and associated file types.

[0052] Step S125 : constructing an initial feature set based on the metadata features, and performing a time series correlation analysis on the initial feature set to determine a dynamic weight coefficient of each feature dimension within a historical storage period.

[0053] Step S126 , performing noise reduction processing on the initial feature set based on the dynamic weight coefficient, removing feature dimensions with weights lower than a preset threshold, and generating an optimized data block feature vector.

[0054] Taking the video data of the video business department as an example, for the video data blocks, the content distribution pattern of each video is analyzed. For example, the color richness of the video screen is determined by analyzing the proportion of different colors in the screen. If the proportion of the main colors such as red, green, and blue in a video screen is relatively uniform, it means that the color richness is high; if a certain color accounts for a large proportion, the color richness is relatively low. At the same time, metadata features are counted, such as the data block size distribution. The data block size of high-definition video may be between 1GB and 5GB, and the data block size of standard-definition video may be between 500MB and 1GB. Hot and cold access tags are determined based on the number of video plays. Data blocks with more than 1,000 plays per week are hot data blocks, and those with less than 100 plays are cold data blocks. Associate file types, such as video subtitle files, cover image files, etc.

[0055] Next, an initial feature set is constructed based on the aforementioned metadata features. For example, the feature set might include "color richness, data block size, hot and cold access tags, and associated file types." A temporal correlation analysis is then performed, using monthly time periods. It was found that certain video types, such as action movies, see a significant increase in playback volume during holidays. Therefore, during this period, the dynamic weight coefficients of the feature dimensions of action movie-related data blocks (such as data block size, hot and cold access tags, etc.) increase. Assume that during non-holiday periods, the weight coefficient for color richness is 0.1, data block size is 0.3, hot and cold access tags is 0.4, and associated file types is 0.2. During holidays, the weight coefficient for color richness, data block size is 0.4, hot and cold access tags is 0.5, and associated file types is 0.05.

[0056] A preset threshold of 0.1 can be set. During the holiday period, the color richness weight coefficient of 0.05 is below the preset threshold and is removed from the feature set. The optimized data block feature vector becomes <data block size, hot and cold access flags, associated file type>, with the weight coefficients becoming 0.44 for data block size (the original 0.4 divided by the sum of the remaining feature dimension weights, 0.9), 0.56 for hot and cold access flags (the original 0.5 divided by 0.9), and 0.0 for associated file type (the original 0.05 was removed because it was below the threshold and is set to 0.0 here to represent the complete vector form).

[0057] For example, for the financial report data of the finance department, the content distribution pattern of each report data block can be analyzed, such as the proportion of different types of data in the report, such as income and expenditure data, assets and liabilities data, etc. Metadata features can be statistically analyzed to determine the size distribution of data blocks. Annual report data blocks may range from 50MB to 100MB, while quarterly report data blocks range from 10MB to 50MB. Hot and cold access tags are determined based on the query frequency of the report. Reports from the most recent quarter are frequently queried, so hot data blocks are considered hot, while reports from many years ago are less frequently queried, so cold data blocks are considered cold. Associated file types, such as report annotation files, can be used.

[0058] Next, we can construct an initial feature set, such as "income and expenditure data ratio, data block size, hot and cold access tags, and associated file types." A quarterly time series correlation analysis reveals that the query frequency for balance sheet-related data blocks increases during the quarterly and year-end financial settlement periods. Therefore, during these periods, the dynamic weight coefficients of the feature dimensions associated with balance sheet-related data blocks (such as data block size and hot and cold access tags) increase. Assume that during the non-settlement period, the weight coefficient for the income and expenditure data ratio is 0.2, the data block size is 0.3, the hot and cold access tags are 0.4, and the associated file type is 0.1. During the settlement period, the weight coefficient for the income and expenditure data ratio changes to 0.1, the data block size changes to 0.4, the hot and cold access tags change to 0.5, and the associated file type changes to 0.0.

[0059] A preset threshold of 0.1 can be set. During the settlement period, if the weight coefficient of the associated file type is 0.0 and falls below the preset threshold, it will be removed from the feature set. The optimized data block feature vector becomes (income and expenditure data ratio, data block size, hot and cold access flags), with the weight coefficients becoming 0.125 for income and expenditure data ratio (0.1 divided by the sum of the remaining feature dimension weights, 0.8), 0.5 for data block size (0.4 divided by 0.8), and 0.375 for hot and cold access flags (0.5 divided by 0.8).

[0060] In one possible implementation, the training method of the storage strategy generation model includes:

[0061] Step S210 , obtaining a historical storage record data set, wherein the historical storage record data set includes a plurality of historical data block feature samples, corresponding historical storage attribute samples and actual storage strategy parameter labels.

[0062] Taking the video data storage of the video business department as an example, for the video business department, the historical storage record dataset includes multiple historical data block feature samples, corresponding historical storage attribute samples and actual storage strategy parameter labels over a period of time.

[0063] Historical data block feature samples include various characteristics of previously stored video data blocks. For example, for different video data block types (HD, SD, different durations, different subject matter, etc.), data block size distribution (such as the frequency distribution of HD video data block sizes between 1GB and 5GB), hot and cold access markers (determining the ratio of hot and cold data blocks based on past playback volume), and associated file types (such as the relationship between subtitle files, cover image files, etc. and the video).

[0064] Historical storage attribute samples cover the previous storage requirements for this video data. In terms of storage capacity distribution, this might record the proportion of video data allocated to various storage nodes over different time periods. In terms of access path optimization, this might record the access latency requirements for videos of varying popularity (hot data blocks and cold data blocks). In terms of data compression levels, this might record the compression levels used for videos with different image quality requirements.

[0065] The "Actual Storage Policy Parameters" tab shows the storage policy parameters actually used for the video data at the time. For example, it includes the actual storage node allocation (which videos are stored on which node), the actual path priority settings (the actual path arrangement to ensure fast access to popular videos), and the actual compression execution parameters (the actual compression ratio for each video).

[0066] Step S220: construct an initial neural network model, which includes a feature interaction layer, a strategy prediction layer, and a constraint verification layer. The constraint verification layer is used to verify whether the prediction strategy parameters meet the configuration requirements of the historical storage attribute samples.

[0067] The feature interaction layer processes input historical data block feature samples and historical storage attribute samples. For example, the feature interaction layer analyzes the relationship between video data block size characteristics and storage capacity distribution attributes to identify storage patterns for large data blocks under different storage capacity distribution requirements.

[0068] The policy prediction layer predicts storage policy parameters based on the results of the feature interaction layer. For example, based on the video's hot and cold access tags and storage capacity distribution, it predicts which storage node a video data block should be assigned to, the priority of its access path, and the compression parameters to use.

[0069] The constraint verification layer verifies whether the policy parameters predicted by the policy prediction layer meet the configuration requirements of the historical storage attribute samples. For example, for video data, if the access delay threshold for popular videos in the historical storage attribute samples requires that transmission start within 1 second (an access path optimization requirement), the constraint verification layer checks whether the delay prediction value corresponding to the path priority parameter predicted by the policy prediction layer meets this upper limit constraint.

[0070] Step S230: Input the historical data block feature samples and the historical storage attribute samples into the initial neural network model, calculate the first loss value between the predicted storage policy parameters output by the policy prediction layer and the actual storage policy parameter labels, and calculate the second loss value between the constraint satisfaction output by the constraint verification layer and the preset standard value.

[0071] Step S240: Adjust the parameters of the initial neural network model according to the weighted sum of the first loss value and the second loss value until the weighted sum is lower than the convergence threshold, thereby obtaining a trained storage strategy generation model.

[0072] In a possible implementation, step S230 includes:

[0073] Step S231: Input the historical data block feature samples and the historical storage attribute samples into the strategy prediction layer of the initial neural network model to generate a set of prediction storage strategy parameters, wherein the set of prediction storage strategy parameters includes prediction node allocation parameters, prediction path priority parameters and prediction compression parameters.

[0074] For example, for a historical video data block, the predicted node allocation parameter may be to allocate it to storage node A, the predicted path priority parameter may be set to high priority (to meet the requirements of fast access to popular videos), and the predicted compression parameter may be to use a compression ratio of 2:1 (assuming that the video quality requirements are high).

[0075] Step S232 , performing item-by-item difference calculation on the predicted storage strategy parameter set and the actual storage strategy parameter label to obtain a node allocation deviation value, a path priority deviation value, and a compression parameter deviation value.

[0076] Assuming that the actual node allocation is storage node B, the node allocation deviation value is the conversion cost of allocating node A to node B (this can be quantified based on factors such as storage node performance differences and data migration costs. For example, the differences between nodes A and B in read and write speed and storage capacity. Assuming that storage node A has a read and write speed of 100 MB / s and node B has a read and write speed of 80 MB / s, and the data block size is 1 GB, then the time cost of migrating data from A to B is 1GB / (80 MB / s) - 1GB / (100 MB / s) = 12.5s - 10s = 2.5s. This 2.5s can be used as part of the node allocation deviation value. Considering storage capacity differences as well, if the remaining capacity of A is 500 GB and the remaining capacity of B is 300 GB, and the risk factor brought by the capacity difference is 0.2, then the node allocation deviation value is 2.5s + 0.2 = 2.7s). Assuming the actual path priority is medium and the predicted path priority is high, the path priority deviation can be calculated based on the latency differences corresponding to different priorities. High priority requires transmission to start within 1 second, while medium priority requires transmission to start within 3 seconds, resulting in a deviation of 2 seconds. Assuming the actual compression ratio is 3:1 and the predicted compression ratio is 2:1, the compression parameter deviation can be calculated based on the difference in compressed data volume. For example, if the original data volume is 1GB, the compression ratio of 2:1 is 0.5GB, and the compression ratio of 3:1 is 0.33GB. The deviation is 0.5GB - 0.33GB = 0.17GB.

[0077] Step S233, according to the configuration requirements in the historical storage attribute sample, assign weight coefficients to the node deviation value, path priority deviation value and compression parameter deviation value, generate a weighted deviation vector, and perform nonlinear normalization processing on the weighted deviation vector to generate the first loss value in scalar form.

[0078] Assuming a weight of 0.3 for storage node assignment (because correct storage node assignment significantly impacts the overall storage architecture), a weight of 0.5 for path priority (access speed is critical to user experience), and a weight of 0.2 for compression (slightly less important than the first two), the weighted deviation vector is 〈2.7s×0.3, 2s×0.5, 0.17GB×0.2〉 = 〈0.81s, 1s, 0.034GB〉. The weighted bias vector is nonlinearly normalized. Assume that the sigmoid function is used for normalization (the calculation process of the sigmoid function is described in detail here. For each element x, sigmoid(x)=1 / (1 + e^(-x)). For 0.81s, sigmoid(0.81) = 1 / (1 + e^(-0.81))≈0.7 is calculated. For 1s, sigmoid(1)=1 / (1 + e^(-1))≈0.73. For 0.034GB, assume that it is first converted to a dimensionless value. For example, it is normalized according to the maximum data volume of the storage system. Assuming the maximum data volume is 10GB, 0.034GB / 10GB=0.0034, sigmoid(0.0034)≈0.5, and the weighted sum of these normalized values ​​is 0.7×0.3+0.73×0.5+0.5×0.2 = 0.675, this 0.675 is the first loss value in scalar form.

[0079] Step S234 , synchronously inputting the predicted storage strategy parameter set and the historical storage attribute sample into the constraint verification layer, parsing the access delay threshold, data persistence level and load balancing coefficient in the historical storage attribute sample.

[0080] For example, for popular videos, the access delay threshold in the historical storage attribute sample is to start transmission within 1 second, the data persistence level is high (with redundant backup and long-term storage), and the load balancing coefficient requires that the usage ratio difference of each storage node does not exceed 10% (hypothesis).

[0081] Step S235: Verify whether the delay prediction value corresponding to the predicted path priority parameter meets the upper limit constraint based on the access delay threshold, and generate the path delay satisfaction; and verify whether the number of redundant copies corresponding to the predicted node allocation parameter meets the minimum redundancy constraint based on the data persistence level, and generate the redundancy satisfaction; and verify whether the node load distribution deviation corresponding to the predicted node allocation parameter is lower than the balance constraint threshold based on the load balancing coefficient, and generate the load balancing satisfaction.

[0082] Assume that the predicted delay value corresponding to the predicted path priority parameter is 0.8 seconds, which meets the requirement of less than 1 second, and the path delay satisfaction is 1 (if the requirement is met, it is set to 1, and if it is not met, it is set to 0). Based on the data persistence level, verify whether the number of redundant copies corresponding to the predicted node allocation parameter meets the minimum redundancy constraint. Assuming that the high level requires 3 redundant copies, the number of redundant copies corresponding to the predicted node allocation parameter is 3, and the redundancy satisfaction is 1. Based on the load balancing coefficient, verify whether the node load distribution deviation corresponding to the predicted node allocation parameter is lower than the balance constraint threshold. Assume that the predicted node allocation parameter allocates the video to node A and node B. The used capacity of node A is 30%, and the used capacity of node B is 32%. The difference is 2%, which is lower than the balance constraint threshold of 10%. The load balancing satisfaction is 1.

[0083] Step S236: Aggregate the path delay satisfaction, redundancy satisfaction, and load balancing satisfaction according to a preset ratio to generate an overall constraint satisfaction, and perform a logarithmic difference calculation between the overall constraint satisfaction and a preset standard value to generate the second loss value.

[0084] Assume the preset proportions are 0.3 for path delay satisfaction, 0.4 for redundancy satisfaction, and 0.3 for load balancing satisfaction. The overall constraint satisfaction is 1 × 0.3 + 1 × 0.4 + 1 × 0.3 = 1. Assume the preset standard value is 1. Calculate the logarithmic difference between the overall constraint satisfaction and the preset standard value. Assuming the logarithmic function log(x) is used, we calculate log(1 / 1) = 0. This 0 is the second loss value.

[0085] Step S237: Input the first loss value and the second loss value into the weight allocator, generate a dynamic weight ratio based on the storage attribute priority of each sample in the historical storage record data set, perform weighted summation on the first loss value and the second loss value based on the dynamic weight ratio, and generate a total loss value.

[0086] Step S238: back-propagate the total loss value to the parameter update module of the initial neural network model, and adjust the trainable parameters of the strategy prediction layer and the constraint verification layer until the total loss value is lower than the convergence threshold.

[0087] The first loss value, 0.675, and the second loss value, 0, are weighted and summed at a certain ratio. Dynamic weighting is generated based on the storage attribute priority of each sample in the historical storage record dataset. Assuming that access speed and storage node allocation are more important in video storage, the weight of the first loss value is 0.6, and the weight of the second loss value is 0.4. The total loss value obtained from this weighted sum is 0.675 × 0.6 + 0 × 0.4 = 0.405.

[0088] The total loss value of 0.405 is backpropagated to the parameter update module of the initial neural network model. At the policy prediction layer, for example, the weights of neurons related to node allocation prediction, path priority prediction, and compression parameter prediction are adjusted. If a neuron is related to the predicted path priority and its weight significantly influences the path priority deviation, its weight is adjusted using a backpropagation algorithm (e.g., gradient descent. The principle of gradient descent is explained in detail here. The weight is adjusted along the negative gradient of the total loss function. Specifically, the weight update amount = - learning rate × derivative of the loss function with respect to the weight. Assuming a learning rate of 0.01, the weight update amount is calculated by calculating the derivative of the loss function with respect to the weight and then multiplying it by -0.01). At the constraint verification layer, trainable parameters related to verifying the access latency threshold, data persistence level, and load balancing coefficient are adjusted. This process is repeated, inputting more historical data block feature samples and historical storage attribute samples, calculating the total loss value, and backpropagating parameter adjustments until the total loss value falls below the convergence threshold (assuming the convergence threshold is 0.01). At this point, the trained storage policy generation model is obtained.

[0089] Furthermore, for the storage of financial statement data in the finance department, the process is similar.

[0090] The historical storage record dataset contains past financial report data block feature samples (such as report data block size, hot and cold access tags, associated file types, etc.), historical storage attribute samples (storage capacity distribution, access path optimization, data compression level, and other requirements), and actual storage policy parameter labels (actual storage node allocation, path priority settings, compression execution parameters, etc.).

[0091] The initial neural network model constructed also has a feature interaction layer, a policy prediction layer, and a constraint verification layer. When calculating the first loss value, for example, the deviation between actual storage node allocation and prediction may be quantified based on differences in storage node security and reliability, as well as data migration costs. Path priority deviation is calculated based on differences in query response time corresponding to different priorities. Compression parameter deviation is calculated based on differences in compressed data accuracy and data volume. These are then weighted and normalized to obtain the first loss value. For the second loss value, the predicted policy parameters are verified based on the access latency threshold (such as the response time requirement for querying recent reports), the data persistence level (such as the long-term retention requirement for annual reports), and the load balancing factor (the requirement for balanced storage node usage) in the historical storage attribute sample. The satisfaction is calculated to obtain the second loss value. Finally, the weighted total loss value is summed and the model parameters are adjusted through backpropagation until convergence.

[0092] In a possible implementation, after step S140, the method further includes:

[0093] Step S150 : collecting actual storage performance indicators of the set of data blocks to be processed in real time. The actual storage performance indicators include node read and write rates, compression efficiency, and path load balance.

[0094] Taking the video business department's video data storage as an example, specialized monitoring tools are used to measure the read and write rates of each storage node storing video data. For example, on storage node A, which stores popular videos, a 1GB block of popular video data is read. The time from issuing the read command to the start of data transmission is recorded. Assuming this time is 0.5 seconds, the read rate is 1GB / 0.5 seconds = 2GB / second. Write rate measurement is similar. When a new video data block (such as a 2GB new video) is to be written to storage node A, the time from the start of the write to its completion is recorded. Assuming this time is 1 second, the write rate is 2GB / 1 second = 2GB / second.

[0095] Calculate the actual compression efficiency. For example, for a video data block originally sized at 3GB, after compression by the storage system, the actual storage size is 1GB. Compression efficiency can be calculated by calculating the ratio of the data volume before and after compression: original size / compressed size = 3GB / 1GB = 3:1.

[0096] Monitor the load of video data on different access paths. Assume that the all-flash storage system has three main access paths: Path 1, Path 2, and Path 3. Count the traffic volume for accessing video data through each path over a period of time (e.g., one hour). If the traffic volume through Path 1 is 500 GB, the traffic volume through Path 2 is 300 GB, and the traffic volume through Path 3 is 200 GB, the path load balance can be calculated using the variance method. First, calculate the average traffic volume: (500 GB + 300 GB + 200 GB) / 3 = 333.33 GB. Then calculate the variance. For Path 1, the variance is 500 GB - 333.33 GB = 166.67 GB; for Path 2, the variance is 300 GB - 333.33 GB = -33.33 GB; and for Path 3, the variance is 200 GB - 333.33 GB = -133.33 GB. The variance is [(166.67GB)² + (- 33.33GB)² + (- 133.33GB)²] / 3 ≈ 11111.11GB² (This calculation is for illustrative purposes only; in practice, a more appropriate load balancing method may be used depending on system characteristics). A smaller value indicates a more balanced load.

[0097] Step S160 : comparing the actual storage performance indicator with the configuration requirements in the expected storage attribute set to generate a performance deviation report.

[0098] In the expected storage attribute set, the read and write rate requirements for popular video storage nodes may be a read rate of no less than 3GB / s and a write rate of no less than 2.5GB / s. However, the actual read rate of storage node A is 2GB / s and the write rate is 2GB / s, both lower than the expected read rate, resulting in a deviation. For compression efficiency, the expected compression ratio may be 2:1, but the actual one is 3:1, which also has a deviation. For path load balancing, the expected variance is no more than 5000GB², but the actual variance is 11111.11GB², which also has a deviation. These deviations are organized into a performance deviation report, which lists in detail the actual value, expected requirements, and deviations for each indicator.

[0099] Step S170: If there are indicator items exceeding the tolerance threshold in the performance deviation report, the online update mechanism of the storage policy generation model is triggered, and the actual storage performance indicator and the corresponding storage request information are input into the model as incremental training data to readjust the model parameters.

[0100] Assume that the tolerance thresholds are: read rate deviation no more than 0.5 GB / s, write rate deviation no more than 0.3 GB / s, compression efficiency deviation no more than 0.5:1, and path load balance variance deviation no more than 3000 GB². Because the actual read rate, write rate, compression efficiency, and path load balance deviations all exceed the tolerance thresholds, the online update mechanism of the storage policy generation model is triggered. Actual storage performance indicators (node ​​read and write rates, compression efficiency, path load balance) and corresponding storage request information (such as video data block type, size, and hot and cold access flags) are fed into the model as incremental training data.

[0101] In a possible implementation, the steps of executing the online update mechanism include:

[0102] Step S171, extract incremental storage operation records within a preset time window from the log database of the all-flash storage system, generate an incremental storage operation record set, and perform field mapping on the storage request information in the incremental storage operation record set and the actual storage performance indicator to generate an incremental training data sample set.

[0103] For example, incremental storage operation records within a preset time window (e.g., the past 24 hours) are extracted from the all-flash storage system's log database. These incremental storage operation records contain information about video data storage operations within that 24-hour period, such as which new video data blocks were stored, and which video data blocks were read or modified. Field mapping is performed between the storage request information (e.g., video data block characteristics) and actual storage performance metrics (e.g., previously collected node read / write rates, compression efficiency, and path load balancing). For example, the video data block size field in the storage request information is mapped to compression efficiency, as block size affects compression efficiency; and the video's hot / cold access flags are mapped to node read / write rates, as popular videos have high read / write frequencies, which impact node read / write rates. This mapping relationship generates a set of incremental training data samples, each containing a set of related storage request information and actual storage performance metrics.

[0104] Step S172 , freezing the network parameters of the feature interaction layer in the storage strategy generation model to generate an initial fine-tuning model in a frozen parameter state.

[0105] The feature interaction layer has already learned some basic relationships between video data block features and storage properties during previous training. To prevent these relationships from being destroyed during fine-tuning, its parameters are frozen. This generates an initial fine-tuning model with frozen parameters. This initial fine-tuning model primarily performs parameter adjustments in the policy prediction layer.

[0106] Step S173: Generate a severity ranking list of indicator items according to the deviation amplitude of each indicator item in the performance deviation report that exceeds the tolerance threshold, and assign layer-by-layer decreasing learning rate weights to the network parameters of the strategy prediction layer based on the severity ranking list of indicator items to generate dynamic learning rate configuration parameters.

[0107] For example, the read rate deviation is 1 GB / s (3 GB / s - 2 GB / s), the write rate deviation is 0.5 GB / s (2.5 GB / s - 2 GB / s), the compression efficiency deviation is 1:1 (2:1 - 3:1), and the path load balance variance deviation is 6111.11 GB² (11111.11 GB² - 5000 GB²). These deviations are sorted by magnitude, assuming the order is path load balance variance deviation, read rate deviation, compression efficiency deviation, and write rate deviation. Based on this ranking of severity, network parameters in the policy prediction layer are assigned descending learning rate weights. Assuming the policy prediction layer has three layers, the top layer, because the path load balance variance deviation is the most severe, is assigned a learning rate weight of 0.01. The middle layer, because the read rate deviation is the second most severe, is assigned a learning rate weight of 0.005. The bottom layer, because the write rate deviation is relatively small, is assigned a learning rate weight of 0.001. This generates the dynamic learning rate configuration parameters.

[0108] Step S174: input the incremental training data sample set into the initial fine-tuning model, perform back-propagation training according to the dynamic learning rate configuration parameters, generate updated strategy prediction layer network parameters, and splice the updated strategy prediction layer network parameters with the feature interaction layer network parameters in the frozen parameter state to generate a candidate updated model parameter set.

[0109] During training, the model predicts policy parameters based on each sample's input (storage request information and actual storage performance metrics). It then calculates the error between the predicted parameters and the actual parameters that should be achieved (based on the desired storage attribute set). Based on this error and the learning rate weight, the network parameters of the policy prediction layer are adjusted. For example, for a video data block in a sample, if the predicted storage node allocation is inappropriate, resulting in a suboptimal read / write rate, the backpropagation algorithm adjusts the network parameters related to storage node allocation in a direction that reduces the error. After multiple rounds of training, the updated policy prediction layer network parameters are generated.

[0110] Step S175: load the candidate update model parameter set into the verification environment of the storage policy generation model, input the verification set storage request information, generate a verification set prediction policy parameter set, calculate the matching degree between the simulated storage performance indicators corresponding to the verification set prediction policy parameter set and the expected storage attribute set in the verification set storage request information, and generate a verification set prediction accuracy improvement value.

[0111] The validation set storage request information is a subset of historical storage request information (for example, 10% of historical video storage request information is selected as the validation set). Based on the candidate update model parameter set, the model generates a validation set prediction policy parameter set. For example, for a video data block storage request in the validation set, the model predicts policy parameters such as storage node allocation, path priority settings, and compression execution parameters. The model then calculates simulated storage performance metrics corresponding to these predicted policy parameter sets, such as predicted node read and write rates, compression efficiency, and path load balancing. The matching degree of these simulated storage performance metrics is calculated against the expected storage attribute set in the validation set storage request information. For example, the matching degree between the simulated node read and write rates and the expected node read and write rates is calculated. If the simulated values ​​meet the expected requirements, the matching degree is 1. Otherwise, the matching degree is calculated based on the deviation (e.g., the larger the deviation, the lower the matching degree). The matching degrees of all validation samples are averaged to obtain the improvement in validation set prediction accuracy.

[0112] Step S176: When the validation set prediction accuracy improvement value exceeds the set percentage, the candidate updated model parameter set is marked as a valid updated parameter set, the valid updated parameter set is encapsulated as a model parameter update instruction, and a synchronization data packet carrying a timestamp and version identifier is generated.

[0113] Step S177, traverse all target storage nodes of the all-flash storage system, detect the difference between the storage policy generation model parameter version currently running on each target storage node and the version identifier in the synchronization data packet, if there is a target storage node running a parameter version earlier than the version identifier in the synchronization data packet, send a forced synchronization signal to the target storage node, trigger the target storage node to interrupt the current storage operation and load the valid update parameter set.

[0114] When the prediction accuracy of the validation set is improved by more than a set percentage (assuming it is set to 10%), the candidate update model parameter set is marked as a valid update parameter set. The valid update parameter set is encapsulated as a model parameter update instruction, and a synchronization data packet carrying a timestamp (recording the time of the update) and a version identifier (indicating that this is a new model version) is generated. All target storage nodes of the all-flash storage system are traversed to detect the difference between the storage policy generation model parameter version currently running on each target storage node and the version identifier in the synchronization data packet. If there is a target storage node running a parameter version earlier than the version identifier in the synchronization data packet, a forced synchronization signal is sent to the target storage node. After receiving the signal, the target storage node interrupts the current storage operation (such as an ongoing video data write or read operation) and loads the valid update parameter set.

[0115] Step S178: After completing parameter synchronization of all target storage nodes, clear outdated data records in the incremental training data sample set to release storage space of the log database.

[0116] For example, data related to incremental storage operation records 24 hours ago is deleted from the incremental training data sample set to release storage space of the log database to provide space for subsequent incremental training data storage.

[0117] The process is similar for the financial department's financial report data storage scenario. When collecting actual storage performance metrics, node read and write rates are measured based on read and write operations on the storage nodes storing the financial report data. Compression efficiency is calculated based on the amount of data before and after compression. Path load balancing is calculated based on the traffic volume on different paths accessing the financial report data. When generating performance deviation reports, these actual metrics are compared with the requirements in the desired storage attribute set (such as read and write rate requirements for fast access to recent reports, compression efficiency requirements for data accuracy, etc.). If any metric exceeds the tolerance threshold, the online update mechanism is triggered, and similar steps are followed, including generating an incremental training data sample set (mapping financial report storage request information with actual storage performance metrics), freezing feature interaction layer network parameters, generating dynamic learning rate configuration parameters, updating the strategy prediction layer network parameters, validating the candidate update model parameter set, updating the model parameters to the target storage node, and cleaning the incremental training data sample set.

[0118] In a possible implementation, when the storage policy generation model is deployed on an edge computing node of an all-flash storage system, the method further includes:

[0119] Step S310: caching frequently used data block feature templates and storage attribute combination patterns locally at the edge computing node.

[0120] In this embodiment, taking the video data storage scenario of the video business department as an example, the edge computing nodes of the video business department's all-flash storage system begin to locally cache frequently used data block feature templates and storage attribute combination patterns. For video data, the data block feature templates may include characteristic information such as video resolution (e.g., HD, SD), video duration range (e.g., less than 5 minutes, 5-30 minutes, longer than 30 minutes), video format (e.g., MP4, AVI), and corresponding storage attribute combination patterns, such as storage capacity distribution (e.g., allocation to specific storage nodes or storage node groups), access path optimization (e.g., setting a specific access priority), and data compression level (e.g., using a specific compression ratio). For example, for a frequently accessed, high-definition, 5-30 minute, MP4-formatted popular video, the corresponding storage attribute combination pattern may be allocation to storage node A with fast read / write speeds, setting a high access priority, and using a low compression ratio (e.g., 2:1). The edge computing node will cache this video data block feature template and storage attribute combination pattern.

[0121] Step S320: When new storage request information is received, the data block feature vector is preferentially matched with the data block feature template in the cache for similarity. If the similarity matching result indicates that the matching degree is higher than the fast response threshold, the pre-generated policy parameters in the cache are directly called to perform the storage operation, bypassing the calculation process of the storage policy generation model.

[0122] When an edge computing node receives a new storage request, such as new video data to be stored, it first performs a similarity match between the data block feature vector of the new video data and the data block feature template in the cache. Assuming the new video is high-definition, 10 minutes long, and in MP4 format, it is compared with the cached high-definition, 5-30 minute, MP4 video data block feature template. When calculating similarity, for the resolution feature, if they are exactly the same, a certain score (e.g., 0.3) is added to the similarity. For the duration range, if the new video falls within the duration range of the cached template, a certain score (e.g., 0.3) is added to the similarity. If the format is the same, a certain score (e.g., 0.3) is added to the similarity, resulting in a total similarity of 0.9. Assuming the fast response threshold is set to 0.8, since 0.9 is higher than 0.8, it indicates a high degree of match. At this time, the pre-generated policy parameters in the cache are directly called for storage operations, that is, the new video is stored according to the previously cached storage attribute combination mode, that is, it is assigned to storage node A, set a high access priority, and adopts a 2:1 compression ratio, thereby bypassing the calculation process of the storage policy generation model and greatly improving the efficiency of the storage operation.

[0123] In a possible implementation, the method for updating the frequently used data block feature template includes:

[0124] Step S410, monitor the policy call records of the edge computing node, obtain the call frequency of each data block feature template and the policy effectiveness generated in the corresponding storage operation, wherein the policy effectiveness is calculated by the matching ratio of the actual performance index after the storage operation and the expected storage attribute set.

[0125] For example, for the HD, 5-30 minute, MP4 video data block feature template, count the number of times it is called within a period (e.g., a day), assuming it is 50 times. This is its call frequency. The policy effectiveness generated by the corresponding storage operation is also calculated. Policy effectiveness is calculated by the matching ratio between the actual performance metrics after the storage operation and the desired storage attribute set. For example, after storing this type of video, the actual storage performance metrics include storage node A's read and write rates, actual compression efficiency, and actual access path load balance. The desired storage attribute set requires storage node A to have a read rate of no less than 3 GB / s, a write rate of no less than 2.5 GB / s, a compression ratio of 2:1, and a path load balance variance of no more than 5000 GB². Actual measurements show that storage node A has a read rate of 3 GB / s, a write rate of 2.5 GB / s, a compression ratio of 2:1, and a path load balance variance of 4000 GB². If all metrics meet the requirements, the policy effectiveness is 1 (because all metrics match, the matching ratio is 100%).

[0126] Step S420 , generating dynamic screening conditions based on the call frequency and policy effectiveness, screening out data block feature templates with a call frequency higher than an active threshold and a policy effectiveness higher than a valid threshold, to form a candidate template set.

[0127] Assume that the active threshold is set to 30 times / day and the effective threshold is set to 0.8. For the high-definition, 5-30 minutes, MP4 video data block feature template, its call frequency is 50 times / day, which is higher than 30 times / day, and the policy effectiveness is 1, which is higher than 0.8, which meets the conditions. All data block feature templates are screened, and those data block feature templates with a call frequency higher than the active threshold and a policy effectiveness higher than the effective threshold are screened out to form a candidate template set. For example, in addition to the above-mentioned video data block feature templates, there is also a standard-definition, shorter than 5 minutes, AVI format video data block feature template with a call frequency of 40 times / day and a policy effectiveness of 0.9. It also meets the conditions and is selected into the candidate template set.

[0128] Step S430 , performing cluster analysis on the data block feature templates in the candidate template set, calculating the similarity between any two data block feature templates, and assigning a weight coefficient to each similarity according to the policy effectiveness, to generate a weighted similarity matrix.

[0129] For example, consider a video data block feature template for HD, 5-30 minutes, MP4, and a video data block feature template for SD, less than 5 minutes, AVI format. When calculating similarity, the resolution feature is reduced by a certain amount (e.g., -0.3) due to the difference between HD and SD. The duration feature is also reduced by a certain amount (e.g., -0.2) due to the difference in duration. The format feature is also reduced by a certain amount (e.g., -0.3), resulting in a combined similarity of 0.2. A weight coefficient is then assigned to each similarity based on the policy effectiveness. Assuming the policy effectiveness of the HD, 5-30 minutes, MP4 video data block feature template is 1, and the policy effectiveness of the SD, less than 5 minutes, AVI format feature template is 0.9. Therefore, the weight coefficient for the HD template is 1 / (1 + 0.9) ≈ 0.53, and the weight coefficient for the SD template is 0.9 / (1 + 0.9) ≈ 0.47. For the similarity of 0.2, the weighted similarity generated based on the weight coefficient is 0.2×0.53 = 0.106 (for the HD template) and 0.2×0.47 = 0.094 (for the SD template). This calculation is performed for all candidate template pairs to generate a weighted similarity matrix.

[0130] Step S440 , traversing the candidate template set based on the weighted similarity matrix, merging the data block feature templates whose similarity is higher than the merging threshold and whose policy effectiveness difference is lower than the tolerance range by feature mean, and generating an optimized merged template set.

[0131] Assume the merge threshold is set to 0.5. If the similarity between two data block feature templates exceeds the merge threshold and the difference in policy effectiveness is below the tolerance range, for example, consider two HD video data block feature templates, one with a duration range of 5-15 minutes and the other with a duration range of 15-30 minutes. Their similarity is 0.6 (above 0.5), and their policy effectiveness is 0.95 and 0.9, respectively. The difference, 0.05, is below the tolerance range (assuming the tolerance range is 0.1). These two templates are then merged using the feature mean. The resolution feature remains unchanged, as both are HD. The duration range is reduced to 5-30 minutes after merging. Other storage attribute combinations, such as storage capacity distribution, access path optimization, and data compression level, remain unchanged if they are the same. If they differ, a weighted average is taken based on the policy effectiveness. For example, if one storage node is assigned to storage node A and the other to storage node B, and the policy effectiveness for storage node A is 0.95 and for storage node B is 0.9, the merged storage node assignment will favor storage node A (calculated based on the weighted average). In this way, an optimized merged template set is generated.

[0132] Step S450: Periodically bidirectionally synchronize the merged template set with the global template library of the central server, and receive a template update instruction issued by the server. The template update instruction includes an identifier of an expired template to be replaced and a corresponding new template in the merged template set.

[0133] Step S460: Clear the data block feature template that matches the expired template identifier in the local cache of the edge computing node according to the template update instruction, insert the new template into the head of the cache queue, and update the values ​​of the active threshold and the effective threshold according to the call frequency distribution of the candidate template set in the most recent synchronization period.

[0134] For example, synchronization occurs once a day. During synchronization, a template update command is received from the server. This command contains the identifier of the expired template to be replaced and the corresponding new template from the merged template set. Suppose the server detects that an old video data block feature template (such as a low-resolution video template with a specific duration range or format) is no longer suitable for current storage needs and marks it as expired. Based on the template update command, the edge computing node removes the data block feature template matching the expired template identifier from the local cache and inserts the new template at the head of the cache queue, giving it priority for matching. The active and effective thresholds are also updated based on the call frequency distribution of the candidate template set during the most recent synchronization cycle. For example, if the call frequency of high-resolution videos has increased recently while the call frequency of low-resolution videos has decreased, the active threshold may be appropriately increased (because high-resolution videos are generally more frequently called). The effective threshold is also adjusted based on the overall policy effectiveness to better meet the video storage needs of the video business department.

[0135] The process is similar for the financial department's financial report data storage scenario. When caching frequently used data block feature templates and storage attribute combinations, these templates may include characteristic information such as report type (e.g., annual report, quarterly report), report size range, report data type (e.g., revenue and expenditure data, balance sheet data), and corresponding storage attribute combinations (e.g., storage node allocation, access path priority, and compression level). When calculating policy call frequency and policy effectiveness, the effectiveness of the policy is calculated based on the matching ratio between the actual performance indicators of the stored financial reports (e.g., storage node read and write rates, compression efficiency, and access path load balancing) and the desired storage attribute set (e.g., access speed requirements for different reports, compression level corresponding to data accuracy requirements). Subsequent operations such as candidate template selection, cluster analysis, template merging, bidirectional synchronization, and template updates are similarly performed based on the characteristics of the financial report data to optimize the storage strategy for financial report data on the edge computing nodes of the all-flash storage system.

[0136] Figure 2 The following diagram illustrates exemplary hardware and software components of an information processing system 100 for all-flash file storage, which can implement the concepts of the present application, according to some embodiments of the present application. For example, the processor 120 can be used in the information processing system 100 for all-flash file storage and perform the functions described in the present application.

[0137] The information processing system 100 for all-flash file storage can be a general-purpose server or a special-purpose server, both of which can be used to implement the information processing method for all-flash file storage of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0138] For example, the information processing system 100 applied to file all-flash storage may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the information processing system 100 applied to file all-flash storage may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The information processing system 100 applied to file all-flash storage also includes an input / output (I / O) interface 150 between the computer and other input / output devices.

[0139] For ease of explanation, only one processor is described in the information processing system 100 applied to all-flash storage of files. However, it should be noted that the information processing system 100 applied to all-flash storage of files in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the information processing system 100 applied to all-flash storage of files executes step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0140] In addition, an embodiment of the present invention further provides a readable storage medium having computer executable instructions preset therein. When a processor executes the computer executable instructions, the above-mentioned information processing method applied to all-flash file storage is implemented.

[0141] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. An information processing method applied to all-flash file storage, characterized in that: The method comprises: Obtaining storage request information received by a target storage node, the storage request information including a set of data blocks to be processed and a corresponding set of desired storage attributes; wherein the set of desired storage attributes is used to describe configuration requirements of the set of data blocks to be processed in terms of storage capacity distribution, access path optimization, and data compression level; Performing multi-dimensional feature extraction on the set of data blocks to be processed to generate a data block feature vector, and performing attribute encoding on the set of expected storage attributes to generate a storage attribute feature vector; Inputting the data block feature vector and the storage attribute feature vector into a pre-trained storage policy generation model, performing dynamic weight fusion on the data block feature vector and the storage attribute feature vector through the storage policy generation model, and outputting a target storage policy parameter set; Generate storage node allocation instructions, path priority configuration and compression execution parameters according to the target storage policy parameter set, and drive the all-flash storage system to perform distributed storage operations on the set of data blocks to be processed based on the storage node allocation instructions; wherein, the target storage policy parameter set meets the constraints of all configuration requirements in the expected storage attribute set.

2. The information processing method for all-flash file storage according to claim 1, characterized in that: The expected storage attribute set includes at least one sub-attribute group, each sub-attribute group corresponds to a storage performance indicator type; and performing attribute encoding on the expected storage attribute set to generate a storage attribute feature vector includes: For each sub-attribute group, select the corresponding encoding rule according to its corresponding storage performance indicator type; If the sub-attribute group is a numerical indicator, it is mapped to a preset dimensional feature space through piecewise normalization; if the sub-attribute group is a logical indicator, a discrete feature sequence is generated through binary vectorization; The feature sequences encoded by each sub-attribute group are weighted and spliced ​​according to the indicator priority to generate a global storage attribute feature vector; wherein the storage performance indicator types include access delay threshold, data persistence level and storage node load balancing coefficient.

3. The information processing method for all-flash file storage according to claim 1, characterized in that: The step of performing multi-dimensional feature extraction on the set of data blocks to be processed to generate a data block feature vector includes: Analyzing the content distribution pattern of each data block in the set of data blocks to be processed, and counting metadata features of each data block, wherein the metadata features include data block size distribution, hot and cold access marks, and associated file types; Constructing an initial feature set based on the metadata features, and performing a time series correlation analysis on the initial feature set to determine a dynamic weight coefficient of each feature dimension within a historical storage period; The initial feature set is subjected to noise reduction processing based on the dynamic weight coefficient, feature dimensions whose weights are lower than a preset threshold are eliminated, and an optimized data block feature vector is generated.

4. The information processing method for all-flash file storage according to claim 3, characterized in that: The training method of the storage strategy generation model includes: Acquire a historical storage record data set, wherein the historical storage record data set includes a plurality of historical data block feature samples, corresponding historical storage attribute samples, and actual storage strategy parameter labels; Constructing an initial neural network model, the initial neural network model comprising a feature interaction layer, a strategy prediction layer, and a constraint verification layer; wherein the constraint verification layer is used to verify whether the prediction strategy parameters meet the configuration requirements of the historical storage attribute samples; Inputting the historical data block feature samples and the historical storage attribute samples into the initial neural network model, calculating a first loss value between the predicted storage policy parameters output by the policy prediction layer and the actual storage policy parameter labels, and calculating a second loss value between the constraint satisfaction output by the constraint verification layer and a preset standard value; The parameters of the initial neural network model are adjusted according to the weighted sum of the first loss value and the second loss value until the weighted sum is lower than a convergence threshold, thereby obtaining a trained storage strategy generation model.

5. The information processing method for all-flash file storage according to claim 4, characterized in that: Inputting the historical data block feature samples and the historical storage attribute samples into the initial neural network model, calculating a first loss value between the prediction parameter output by the strategy prediction layer and the actual storage strategy parameter label, and calculating a second loss value between the constraint satisfaction output by the constraint verification layer and a preset standard value, includes: Inputting the historical data block feature samples and the historical storage attribute samples into the strategy prediction layer of the initial neural network model to generate a set of prediction storage strategy parameters, wherein the set of prediction storage strategy parameters includes a prediction node allocation parameter, a prediction path priority parameter, and a prediction compression parameter; Calculate the difference between the predicted storage strategy parameter set and the actual storage strategy parameter label item by item to obtain the node allocation deviation value, the path priority deviation value and the compression parameter deviation value; According to the configuration requirements in the historically stored attribute sample, assigning weight coefficients to the node deviation value, the path priority deviation value, and the compression parameter deviation value to generate a weighted deviation vector, and performing nonlinear normalization processing on the weighted deviation vector to generate the first loss value in scalar form; Synchronously inputting the predicted storage strategy parameter set and the historical storage attribute sample into the constraint verification layer, parsing the access delay threshold, data persistence level and load balancing coefficient in the historical storage attribute sample; Verifying whether the delay prediction value corresponding to the predicted path priority parameter satisfies the upper limit constraint based on the access delay threshold, and generating a path delay satisfaction degree; and verifying whether the number of redundant copies corresponding to the predicted node allocation parameter satisfies the minimum redundancy constraint based on the data persistence level, and generating a redundancy satisfaction degree; and verifying whether the node load distribution deviation corresponding to the predicted node allocation parameter is lower than the balance constraint threshold based on the load balancing coefficient, and generating a load balancing satisfaction degree; Aggregating the path delay satisfaction, redundancy satisfaction, and load balancing satisfaction according to a preset ratio to generate an overall constraint satisfaction, and performing a logarithmic difference calculation between the overall constraint satisfaction and a preset standard value to generate the second loss value; Inputting the first loss value and the second loss value into a weight allocator, generating a dynamic weight ratio according to the storage attribute priority of each sample in the historical storage record data set, and performing a weighted summation of the first loss value and the second loss value according to the dynamic weight ratio to generate a total loss value; The total loss value is back-propagated to the parameter update module of the initial neural network model, and the trainable parameters of the strategy prediction layer and the constraint verification layer are adjusted until the total loss value is lower than the convergence threshold.

6. The information processing method for all-flash file storage according to claim 5, characterized in that: After driving the all-flash storage system to perform a distributed storage operation on the set of data blocks to be processed based on the storage node allocation instruction, the method further includes: Collecting actual storage performance indicators of the set of data blocks to be processed in real time, wherein the actual storage performance indicators include node read and write rate, compression efficiency, and path load balance; Comparing the actual storage performance indicator with the configuration requirements in the expected storage attribute set to generate a performance deviation report; If there are indicator items in the performance deviation report that exceed the tolerance threshold, the online update mechanism of the storage policy generation model is triggered, and the actual storage performance indicators and corresponding storage request information are input into the model as incremental training data to readjust the model parameters.

7. The information processing method for all-flash file storage according to claim 6, characterized in that: The execution steps of the online update mechanism include: Extracting incremental storage operation records within a preset time window from the log database of the all-flash storage system to generate an incremental storage operation record set, and performing field mapping between storage request information in the incremental storage operation record set and actual storage performance indicators to generate an incremental training data sample set; Freezing network parameters of the feature interaction layer in the storage strategy generation model to generate an initial fine-tuning model in a frozen parameter state; Generate a ranking list of indicator item severity based on the deviation magnitude of each indicator item in the performance deviation report that exceeds the tolerance threshold, and assign layer-by-layer decreasing learning rate weights to the network parameters of the strategy prediction layer based on the ranking list of indicator item severity to generate dynamic learning rate configuration parameters; Inputting the incremental training data sample set into the initial fine-tuning model, performing backpropagation training according to the dynamic learning rate configuration parameters to generate updated policy prediction layer network parameters, and splicing the updated policy prediction layer network parameters with the feature interaction layer network parameters in the frozen parameter state to generate a candidate updated model parameter set; The candidate update model parameter set is loaded into the verification environment of the storage policy generation model, the verification set storage request information is input, a verification set prediction policy parameter set is generated, and a matching degree calculation is performed between the simulated storage performance index corresponding to the verification set prediction policy parameter set and the expected storage attribute set in the verification set storage request information to generate a verification set prediction accuracy improvement value; When the prediction accuracy of the validation set is improved by more than a set percentage, the candidate updated model parameter set is marked as a valid updated parameter set, the valid updated parameter set is encapsulated as a model parameter update instruction, and a synchronization data packet carrying a timestamp and version identifier is generated; Traversing all target storage nodes of the all-flash storage system, detecting the difference between the storage policy generation model parameter version currently running on each target storage node and the version identifier in the synchronization data packet, and if there is a target storage node running a parameter version earlier than the version identifier in the synchronization data packet, sending a forced synchronization signal to the target storage node, triggering the target storage node to interrupt the current storage operation and load the valid update parameter set; After completing parameter synchronization of all target storage nodes, expired data records in the incremental training data sample set are cleared to release storage space of the log database.

8. The information processing method for all-flash file storage according to claim 1, characterized in that: When the storage strategy generation model is deployed on an edge computing node of an all-flash storage system, the method further includes: Cache frequently used data block feature templates and storage attribute combination patterns locally at edge computing nodes; When new storage request information is received, the data block feature vector is preferentially matched with the data block feature template in the cache for similarity. If the similarity matching result indicates that the matching degree is higher than the fast response threshold, the pre-generated policy parameters in the cache are directly called to perform the storage operation, bypassing the calculation process of the storage policy generation model.

9. The information processing method for all-flash file storage according to claim 8, characterized in that: The method for updating the frequently used data block feature template includes: Monitor the policy call records of edge computing nodes to obtain the call frequency of each data block feature template and the policy effectiveness generated in the corresponding storage operation, where the policy effectiveness is calculated by the matching ratio between the actual performance index after the storage operation and the expected storage attribute set; Generate dynamic screening conditions based on the call frequency and policy effectiveness, filter out data block feature templates whose call frequency is higher than the active threshold and whose policy effectiveness is higher than the valid threshold, and form a candidate template set; Performing cluster analysis on the data block feature templates in the candidate template set, calculating the similarity between any two data block feature templates, and assigning a weight coefficient to each similarity according to the effectiveness of the strategy to generate a weighted similarity matrix; Traversing the candidate template set based on the weighted similarity matrix, merging the feature mean of the data block feature templates whose similarity is higher than the merging threshold and whose policy effectiveness difference is lower than the tolerance range, to generate an optimized merged template set; Periodically bidirectionally synchronizing the merged template set with the global template library of the central server, and receiving a template update instruction issued by the server, the template update instruction including an identifier of an expired template to be replaced and a corresponding new template in the merged template set; According to the template update instruction, the data block feature template matching the expired template identifier in the local cache of the edge computing node is cleared, and the new template is inserted into the head of the cache queue. At the same time, the values ​​of the active threshold and the effective threshold are updated according to the call frequency distribution of the candidate template set in the most recent synchronization cycle.

10. An information processing system for all-flash file storage, characterized in that: The information processing system applied to all-flash file storage includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the information processing method applied to all-flash file storage as described in any one of claims 1 to 9 above.

Images