Storage optimization method and device based on seismic data processing and medium

Abstract

The invention provides a storage optimization method and device based on seismic data processing and a medium, and belongs to the field of computers. The method comprises the following steps: collecting log information through a data access log, analyzing and converting the log information into structured data, and performing distributed storage; extracting spatial features and time features of the structured data, and fusing the spatial features and the time features to obtain fused features; constructing a deep learning model by adopting the fused features, and training to obtain a prediction model; predicting data blocks, time and frequency accessed in the future according to the output of the prediction model; according to a prediction result, required data blocks are extracted and stored in a rapid storage or cache; the method comprises the following steps: preprocessing and initializing data blocks in a quick storage or cache, and then carrying out hierarchical storage. According to the method, accurate prediction of the next task data demand is realized, and core information such as data blocks, access time and frequency is covered, so that preprocessing and storage optimization strategies of data are effectively guided.

Description

Technical Field

[0001] This invention belongs to the field of computer science, and specifically relates to a storage optimization method, apparatus, and medium based on seismic data processing. Background Technology

[0002] With the continuous increase in exploration and development activities, the demand for computationally intensive geophysical technologies in seismic data processing is also growing. Since 2005, the density of seismic data acquisition channels has increased significantly, surging from 200,000 channels per square kilometer to over 1 million channels today. Controlled-source acquisition projects, in particular, have a massive data volume, averaging 453 GB per square kilometer. In comparison, explosive-source acquisition projects have a data volume of only 39 GB per square kilometer, while controlled-source acquisition projects combining well and shot methods have a data volume of 103 GB per square kilometer. The data volume of controlled-source seismic data is approximately 11.6 times that of explosive-source data; correspondingly, the computing and storage capabilities required to process this data are typically more than 10 times that of conventional data. The application of high-precision 3D seismic acquisition technology places higher demands on the capacity, performance, scalability, and reliability of storage systems. Faced with such massive amounts of data, traditional general-purpose storage systems are struggling to meet the high-speed data I / O requirements, making the improvement of I / O rates one of the key bottlenecks that urgently needs to be addressed.

[0003] Current data retrieval methods primarily rely on the NFS (Network File System) protocol to access distributed storage. NFS, a distributed file system protocol pioneered by Sun Microsystems in 1984, offers the core advantage of allowing clients to seamlessly access file systems on remote servers over a network. Built on TCP / IP networks, this protocol provides clients with a transparent access mechanism, enabling users to access remote file systems as easily as if they were operating a local file system, without needing to concern themselves with the actual storage location of files or the complex underlying network transmission details. This design greatly simplifies the file access process and improves the flexibility and efficiency of data processing.

[0004] In seismic data processing, using the NFS protocol to map file systems to remote servers over the network presents two major challenges. First, there are network latency and bandwidth limitations. Since both read and write operations require data transmission over the network, unstable network performance or limited bandwidth can negatively impact processing performance. Second, in scenarios like seismic processing that require large-scale concurrent data access, the NFS client / server architecture may cause uneven storage I / O load, thus affecting overall processing performance. Finally, under high concurrency, the processing capacity of the NFS server may become a bottleneck. The server needs to handle a large number of network requests and file operations; if the server performance is insufficient, it will be unable to meet the access demands of clients. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in the prior art and provide a storage optimization method, device and medium based on seismic data processing. By using deep learning technology, the business storage characteristics of seismic data processing are modeled to achieve accurate prediction of the data requirements of the next task. Based on the prediction results, the system can deliver the data to the storage system for initialization processing in advance, thereby effectively overcoming the problems of network latency and bandwidth limitation and significantly improving the overall performance of seismic data processing.

[0006] This invention is achieved through the following technical solution:

[0007] A first aspect of the present invention provides a storage optimization method based on seismic data processing, comprising the following steps:

[0008] Step 100: Collect log information through data access logs, parse the log information and convert it into structured data, and then store it in a distributed manner;

[0009] Step 200: Extract the spatial and temporal features of the structured data respectively, and fuse the spatial and temporal features to obtain the fused features;

[0010] Step 300: Construct a deep learning model using the fused features and train it to obtain a prediction model;

[0011] Step 400: Based on the output of the prediction model, predict the data blocks, time, and frequency of future accesses;

[0012] Step 500: Based on the prediction results, extract the required data blocks from the distributed storage and store them in fast storage or cache;

[0013] Step 600: After preprocessing and initializing the data blocks in the fast storage or cache, perform tiered storage.

[0014] A further improvement of the present invention is that:

[0015] The log information is parsed and transformed into structured data, then stored in a distributed manner. Specific operations include:

[0016] Step 101: Preprocess the collected log information to remove duplicate data, process abnormal data, and merge related log entries;

[0017] Step 102: Parse the preprocessed log information;

[0018] Step 103: Map the parsed log information to the target data structure, convert the unstructured log information into a structured data format and store it in a distributed manner.

[0019] A further improvement of the present invention is that

[0020] Step 200: Extract the spatial and temporal features of the structured data respectively, and fuse the spatial and temporal features to obtain the fused features. Specific operations include:

[0021] Step 201: Use a convolutional neural network (CNN) to process the structured data to obtain the spatial feature vector F_spatial;

[0022] Step 202: The structured data is processed using a Long Short-Term Memory (LSTM) network to obtain the temporal feature vector F_temporal;

[0023] Step 203: The spatial feature vector F_spatial and the temporal feature vector F_temporal are fused together using feature concatenation or weighted averaging to obtain the fused feature vector F_fused = [F_spatial, F_temporal].

[0024] A further improvement of the present invention is that:

[0025] In step 300, the model training is performed using an adaptive optimization algorithm.

[0026] A further improvement of the present invention is that:

[0027] Step 600 involves preprocessing and initializing the data blocks in the fast storage or cache. Specific operations include:

[0028] (1) Clean and convert the format of the extracted data blocks;

[0029] (2) Establish an index structure for the preprocessed data blocks;

[0030] (3) Load metadata information for data blocks with established index structures.

[0031] A further improvement of the present invention is that:

[0032] Use B-trees or hash tables to build the index structure for data blocks.

[0033] A further improvement of the present invention is that:

[0034] Metadata information includes access permissions and storage paths for data blocks.

[0035] A further improvement of the present invention is that:

[0036] Step 600 involves tiered storage, specifically including the following operations:

[0037] Based on the data block's popularity level, its distribution across different storage media is dynamically adjusted: hot data is stored in high-speed SSDs or memory; warm data is stored in medium-speed HDDs; and cold data is stored in low-speed storage media.

[0038] A second aspect of the present invention provides a storage optimization apparatus based on seismic data processing, comprising:

[0039] The data acquisition and parsing unit is used to collect log information through data access logs, parse the log information to convert it into structured data, and store it in a distributed manner.

[0040] The feature extraction and fusion unit is used to extract the spatial and temporal features of the structured data respectively, and to fuse the spatial and temporal features to obtain the fused features;

[0041] The model building unit is used to build a deep learning model using the fused features, and then train it to obtain a prediction model.

[0042] The prediction unit is used to predict the data blocks, times, and frequencies to be accessed in the future based on the output of the prediction model.

[0043] The data block extraction unit is used to extract the required data blocks from the distributed storage based on the prediction results and store them in fast storage or cache;

[0044] Tiered storage units are used to store data blocks in fast storage or cache in tiers after preprocessing and initialization.

[0045] A third aspect of the present invention provides a computer-readable storage medium storing at least one computer-executable program, which, when executed by the computer, causes the computer to perform the steps in the storage optimization method based on seismic data processing.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] This invention utilizes deep learning technology to deeply analyze the operational storage characteristics during seismic data processing, including key factors such as data access patterns, data popularity, and concurrent access volume, and models these characteristics to capture data access patterns, laying a solid foundation for subsequent data demand prediction. Based on this model, this invention achieves accurate prediction of data requirements for the next task, covering core information such as data blocks, access time, and frequency, thereby effectively guiding data preprocessing and storage optimization strategies.

[0048] This invention, on the one hand, can reduce the impact of network latency and bandwidth limitations. Because the system can predict and retrieve the data blocks required for the next task to the local or nearby storage system, the latency and bandwidth consumption of data access are significantly reduced during actual task execution, thereby improving the speed and efficiency of seismic data processing. On the other hand, it enhances the overall performance of seismic data processing. By accurately predicting data requirements and performing preprocessing and initialization, the system can better meet the data access needs during seismic data processing, reducing the computational idle time caused by data waiting, thereby improving the overall processing performance. Attached Figure Description

[0049] Figure 1 This is a flowchart of a storage optimization method based on seismic data processing in an embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings:

[0051]

Example 1

[0052] This invention provides a storage optimization method based on seismic data processing, such as... Figure 1 As shown, the specific steps include:

[0053] Step 100: Collect log information through data access logs, parse the log information and convert it into structured data, and then store it in a distributed manner;

[0054] Step 200: Extract the spatial and temporal features of the structured data respectively, and fuse the spatial and temporal features to obtain the fused features;

[0055] Step 300: Construct a deep learning model using the fused features and train it to obtain a prediction model;

[0056] Step 400: Based on the output of the prediction model, predict the data blocks, time, and frequency of future accesses;

[0057] Step 500: Based on the prediction results, extract the required data blocks from the distributed storage and store them in fast storage or cache;

[0058] Step 600: After preprocessing and initializing the data blocks in the fast storage or cache, perform tiered storage.

[0059] This invention utilizes deep learning technology to deeply analyze the operational storage characteristics during seismic data processing, including key factors such as data access patterns, data popularity, and concurrent access volume, and models these characteristics to capture data access patterns, laying a solid foundation for subsequent data demand prediction. Based on this model, this invention achieves accurate prediction of data requirements for the next task, covering core information such as data blocks, access time, and frequency, thereby effectively guiding data preprocessing and storage optimization strategies.

[0060] This invention, on the one hand, can reduce the impact of network latency and bandwidth limitations. Because the system can predict and retrieve the data blocks required for the next task to the local or nearby storage system, the latency and bandwidth consumption of data access are significantly reduced during actual task execution, thereby improving the speed and efficiency of seismic data processing. On the other hand, it enhances the overall performance of seismic data processing. By accurately predicting data requirements and performing preprocessing and initialization, the system can better meet the data access needs during seismic data processing, reducing the computational idle time caused by data waiting, thereby improving the overall processing performance.

[0061]

Example 2

[0062] Step 100: Collect log information through data access logs, parse the log information to transform it into structured data, and then store it in a distributed manner.

[0063] Log information includes data block ID, access frequency, timestamp, access source, access device type, and access duration.

[0064] The log information is parsed and transformed into structured data. Specific operations include:

[0065] Step 101: Preprocess the collected log information, such as removing duplicate data, handling abnormal data, merging related log entries, etc.

[0066] Step 102 involves parsing the preprocessed log information, specifically as follows:

[0067] For logs with a relatively fixed format, regular expressions are used to match and extract key information. This method is suitable for situations where the log format is relatively standardized and does not change much.

[0068] For logs that use structured formats such as JSON or XML, use the corresponding parser to parse the logs and extract the required fields;

[0069] For logs with complex formats, write custom parsing rules or scripts to parse the logs;

[0070] Step 103: Map the parsed log information to the target data structure, convert the unstructured log information into a structured data format and distribute it to the database for subsequent data processing and analysis.

[0071] Data access pattern analysis is key to understanding and optimizing storage systems. It helps identify the behavioral characteristics of data access, providing support for subsequent storage optimization and data scheduling.

[0072] In this embodiment of the invention, common data access sequences and patterns are identified by analyzing the data access logs of historical processing tasks, including:

[0073] Access start time: Recording the start time of each data access can help identify peak and off-peak periods of access volume throughout the day.

[0074] Access duration: Analyze the access duration of each data block to identify user behavior patterns.

[0075] Access frequency: Calculate the number of times a specific data block is accessed within a certain period of time to identify frequently accessed data blocks.

[0076] In this embodiment of the invention, specialized automated tools are used to mine log data, employing frequent pattern mining algorithms to identify relationships and common access patterns between data blocks. These tools can automatically identify frequent access sequences between data, thereby revealing potential patterns in data access.

[0077] By performing time series analysis on data access sequences, periodicity or regularity of patterns can be identified. Time series analysis techniques (such as ARIMA and Prophet) can be used to explore the periodic fluctuations in data access.

[0078] By statistically analyzing the access frequency distribution of different data blocks, access hotspots can be identified. These densely accessed areas can be visualized by generating histograms or heatmaps of access frequencies. Identifying access hotspots helps determine which data blocks should be prioritized for caching or storage on fast media. Specifically, the matplotlib or seaborn libraries in Python can be used to generate frequency charts of data access. These tools offer rich visualization capabilities to help visually represent access patterns.

[0079] By analyzing access time intervals, data blocks with high access frequency and short access intervals can be identified and allocated higher priority storage resources. Analysis tools: Pandas DataFrames are used for data processing and analysis to calculate statistical indicators and generate analysis reports.

[0080] To allocate storage resources effectively and improve system response speed, it is necessary to assess the popularity of data blocks. For example, the following methods can be used:

[0081] Counter mechanism: An access counter is maintained for each data block to record the number of times it has been accessed. The counter is updated in real time to reflect the latest access status. Specifically, the access counter can be maintained in an in-memory database such as Redis or Memcached, leveraging its fast read and write capabilities to maintain the real-time performance of the counter. After each access, the counter value is updated through an auto-increment operation and periodically persisted to disk storage to prevent data loss.

[0082] Time window selection: Set an appropriate time window (such as the past week), count the number of accesses to each data block within that window, and evaluate its popularity changes. By setting different time windows, the system can analyze short-term and long-term changes in data popularity.

[0083] Short-term and long-term popularity analysis: Short-term popularity can guide immediate caching strategies, while long-term popularity helps optimize the overall storage architecture.

[0084] Sliding window mechanism: Using the sliding window technique, popularity statistics can be continuously updated, maintaining the real-time nature and accuracy of the analysis. This paper implements sliding window data stream analysis using Python's `collections.deque`.

[0085] Based on the number of times data blocks are accessed, they are divided into different popularity levels:

[0086] Very hot: Accessed very frequently, requires persistent resident cache to improve access efficiency;

[0087] Warm: Accessed frequently, suitable for storage on high-speed storage media;

[0088] Lukewarm: Access frequency is generally low, so storage on medium-speed media can be considered;

[0089] Cold: Accessed very infrequently, suitable for storage on slow media to save resources.

[0090] Implementation details: After each time window ends, the system updates the popularity level of the data block based on the access count. Access counts can be batch-processed using SQL queries, and the storage strategy for the data blocks can be adjusted.

[0091] The system periodically reassesses the popularity level of data blocks and dynamically adjusts their storage strategies based on the latest access records. This ensures efficient utilization of storage resources and enables rapid response to changes in data access demands. Data popularity assessments and storage strategy updates are performed regularly using Python scripts or scheduling tools such as Apache Airflow.

[0092]

Example 3

[0093] Step 200: Extract the spatial and temporal features of the structured data respectively, and fuse the spatial and temporal features to obtain the fused features. Specific operations include:

[0094] Step 201: Use a convolutional neural network (CNN) to process the structured data to obtain the spatial feature vector F_spatial;

[0095] Step 202: The structured data is processed using a Long Short-Term Memory (LSTM) network to obtain the temporal feature vector F_temporal;

[0096] Step 203: The spatial feature vector F_spatial and the temporal feature vector F_temporal are fused together using feature concatenation or weighted averaging to obtain the fused feature vector F_fused = [F_spatial, F_temporal].

[0097] To predict future data access trends and optimize storage strategies, this invention combines Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) technologies to model and analyze data access characteristics.

[0098] Convolutional Neural Networks (CNNs) excel at processing and extracting spatial features from data. Through convolution and pooling operations, CNNs can identify access patterns and spatial distributions of data blocks.

[0099] Convolutional layers: These layers extract local features from the data through convolution operations, identifying relationships and common access patterns between data blocks. These features can include local hotspots and coldspots in data access.

[0100] Pooling layers: These downsample features, reducing computational cost and feature dimensionality while preserving key pattern information. Pooling operations can effectively reduce the size of feature maps and improve computational efficiency.

[0101] Pooling strategy: Use max pooling or average pooling for downsampling, and choose a suitable data dimensionality reduction strategy to retain important information.

[0102] A CNN model is built using `tf.keras.layers.Conv2D` and `tf.keras.layers.MaxPooling2D` from TensorFlow to extract spatial features. Spatial features include:

[0103] Data block ID: A unique identifier that identifies a data block and is used to distinguish different data objects;

[0104] Storage location: The physical or logical location of a data block, reflecting the distribution of data in the storage system.

[0105] Access frequency: The number of times each data block is accessed within a specific time window is an important basis for assessing the popularity of data.

[0106] Based on the spatial features extracted by CNN, the distribution of hot and cold data is distinguished. The system can adopt different storage and access strategies for data blocks with different levels of popularity; for example, hot data can be cached in faster storage media. Strategy optimization: Using the features extracted by CNN, an automated algorithm classifies data blocks into hot, warm, and cold categories and formulates corresponding storage strategies. This strategy optimization can significantly improve the system's response speed and resource utilization efficiency.

[0107] Long Short-Term Memory (LSTM) networks excel at capturing long-term dependencies and dynamic changes in time-series data, and model access trends through their unique gating mechanism.

[0108] LSTM's memory cells store and update historical information, capturing trends in access patterns. Through input gates, forget gates, and output gates, LSTM effectively manages and updates memory information. Implementation: LSTM layers are created using `tf.keras.layers.LSTM` in `TensorFlow` to analyze the time-series characteristics of data access.

[0109] LSTM can predict future data access trends and patterns based on the analysis of historical data. This helps to adjust storage strategies and optimize data scheduling in advance, store relevant data in the cache in advance, reduce access latency and improve system response speed.

[0110] This invention combines the temporal features extracted by LSTM with the spatial features extracted by CNN to construct a complete feature vector. The combined feature vector can comprehensively characterize the spatiotemporal characteristics of data access.

[0111] Fusion strategy: Use feature concatenation or weighted averaging to fuse spatial and temporal features into a unified vector input, ensuring the synergistic effect of the model in capturing features of different dimensions.

[0112] Implementation tools: Feature fusion is performed using `tf.keras.layers.Concatenate` from `TensorFlow` to facilitate subsequent model training and optimization.

[0113]

Example 4

[0114] Step 300: Construct a deep learning model using the fused features and train it to obtain the prediction model.

[0115] In this embodiment of the invention, the fused features are used for modeling to optimize the access strategy of the storage system and improve the prediction accuracy of data demand. A hybrid model can be constructed by combining CNN and LSTM to fully utilize the advantages of both networks. Through model optimization, the system can dynamically adjust storage and caching strategies under different access scenarios.

[0116] In this embodiment of the invention, the high-level API provided by `TensorFlow` is used to build, train, and validate deep learning models, ensuring the efficiency and scalability of the models.

[0117] In this embodiment of the invention, adaptive optimization algorithms such as `Adam` or `RMSProp` are used for model training to improve the convergence speed and prediction accuracy of the model.

[0118] In this embodiment of the invention, cross-validation and metric evaluation (such as MAE, RMSE) are used to evaluate and validate the model to ensure the model's generalization ability and stability.

[0119]

Example 5

[0120] Step 400: Based on the output of the prediction model, predict the data blocks, time, and frequency of future accesses.

[0121] The predictive model output will include predictions of the data blocks to be accessed in the future, the access time (or time interval), and the access frequency. The system can prepare cache and storage resources in advance and optimize data access strategies based on this predictive information.

[0122] Predict accessed data blocks: Identify the IDs of data blocks that are about to be accessed and prepare cache and storage resources for them in advance.

[0123] Predict access time: Predict the time of future access and prepare data prefetching and caching in advance.

[0124] Predict access frequency: Predict the frequency of accesses over a future period to facilitate the rational allocation of cache and storage resources.

[0125] Storage Location: Predict the optimal storage location for data blocks to facilitate fast access. Dynamically adjust the storage location based on data access frequency and intensity to optimize system performance.

[0126]

Example 6

[0127] Step 500: Based on the prediction results, extract the required data blocks from the distributed storage and store them in fast storage or cache.

[0128] Based on the prediction results, the system can prefetch the required data blocks from the slow data medium in the distributed storage to the fast storage medium or cache in advance. This prefetching strategy can significantly reduce the latency of data access.

[0129] In this embodiment of the invention, the cache content and strategy are dynamically adjusted to ensure that frequently accessed data is cached first.

[0130] Caching strategy: A dynamic caching strategy is adopted, prioritizing the storage of frequently accessed data in the high-speed cache to reduce access latency. Through caching strategies such as LRU (Least Recently Used) and LFU (Least Frequently Used), the system can flexibly adjust cached content.

[0131]

Example 7

[0132] Step 600 involves preprocessing and initializing the data blocks in the fast storage or cache, followed by tiered storage.

[0133] Preprocessing and initializing data blocks in fast storage or cache, specifically including:

[0134] (1) Preprocessing operations such as cleaning and format conversion on the pulled data blocks help improve data quality and consistency. Cleaning and format conversion usually use existing mature methods and technologies, which will not be elaborated here.

[0135] (2) Establish an index structure for the preprocessed data blocks.

[0136] To accelerate data retrieval, an index structure for data blocks is established using data structures such as B-trees or hash tables to enable rapid location and access of data blocks. These index structures can be optimized based on data access patterns and characteristics. Index implementation tools utilize search engine technologies such as Elasticsearch or Lucene to build and optimize the index structure of data blocks, thereby improving data retrieval efficiency.

[0137] (3) Load metadata information for data blocks with established index structures. The metadata information includes the data block's access permissions and storage path to ensure that the data block can be quickly located and accessed during task execution. By preloading metadata information, the system can quickly obtain relevant information at the start of the task.

[0138] The tiered storage mechanism includes the following specific operations:

[0139] Based on the data block's popularity level, its distribution across different storage media is dynamically adjusted. Hot data is stored in high-speed SSDs or memory to improve access speed; warm data is stored in medium-speed HDDs to balance access speed and cost; and cold data is stored in low-speed storage media (such as magnetic tape) to save costs.

[0140] This invention utilizes distributed storage systems such as Ceph or GlusterFS to implement intelligent scheduling and tiered storage strategies, optimizing resource utilization. Through intelligent scheduling and load balancing, storage resource usage is optimized, avoiding bottlenecks and waste. Using load balancing algorithms, the system can rationally distribute data across multiple storage devices, preventing overload of any single device.

[0141]

Example 8

[0142] This invention provides a storage optimization device based on seismic data processing, comprising:

[0143] The data acquisition and parsing unit is used to collect log information through data access logs, parse the log information to convert it into structured data, and store it in a distributed manner.

[0144] The feature extraction and fusion unit is used to extract the spatial and temporal features of the structured data respectively, and to fuse the spatial and temporal features to obtain the fused features;

[0145] The model building unit is used to build a deep learning model using the fused features, and then train it to obtain a prediction model.

[0146] The prediction unit is used to predict the data blocks, times, and frequencies to be accessed in the future based on the output of the prediction model.

[0147] The data block extraction unit is used to extract the required data blocks from the distributed storage based on the prediction results and store them in fast storage or cache;

[0148] Tiered storage units are used to store data blocks in fast storage or cache in tiers after preprocessing and initialization.

[0149]

Example 9

[0150] This invention provides a computer-readable storage medium storing at least one computer-executable program. When executed by the computer, the at least one program causes the computer to perform steps in the storage optimization method based on seismic data processing. The steps include:

[0151] Step 100: Collect log information through data access logs, parse the log information and convert it into structured data, and then store it in a distributed manner;

[0152] Step 200: Extract the spatial and temporal features of the structured data respectively, and fuse the spatial and temporal features to obtain the fused features;

[0153] Step 300: Construct a deep learning model using the fused features and train it to obtain a prediction model;

[0154] Step 400: Based on the output of the prediction model, predict the data blocks, time, and frequency of future accesses;

[0155] Step 500: Based on the prediction results, extract the required data blocks from the distributed storage and store them in fast storage or cache;

[0156] Step 600: After preprocessing and initializing the data blocks in the fast storage or cache, perform tiered storage.

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0158] The above technical solution is only one embodiment of the present invention. For those skilled in the art, based on the principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the technical solutions described in the specific embodiments of the present invention. Therefore, the foregoing description is only a preferred option and is not restrictive.

Claims

1. A storage optimization method based on seismic data processing, characterized in that, Includes the following steps: Step 100: Collect log information through data access logs, parse the log information and convert it into structured data, and then store it in a distributed manner; Step 200: Extract the spatial and temporal features of the structured data respectively, and fuse the spatial and temporal features to obtain the fused features; Step 300: Construct a deep learning model using the fused features and train it to obtain a prediction model; Step 400: Based on the output of the prediction model, predict the data blocks, time, and frequency of future accesses; Step 500: Based on the prediction results, extract the required data blocks from the distributed storage and store them in fast storage or cache; Step 600: After preprocessing and initializing the data blocks in the fast storage or cache, perform tiered storage.

2. The method according to claim 1, characterized in that, The log information is parsed and transformed into structured data, then stored in a distributed manner. Specific operations include: Step 101: Preprocess the collected log information to remove duplicate data, process abnormal data, and merge related log entries; Step 102: Parse the preprocessed log information; Step 103: Map the parsed log information to the target data structure, convert the unstructured log information into a structured data format and store it in a distributed manner.

3. The method according to claim 1, characterized in that, Step 200: Extract the spatial and temporal features of the structured data respectively, and fuse the spatial and temporal features to obtain the fused features. Specific operations include: Step 201: Use a convolutional neural network (CNN) to process the structured data to obtain the spatial feature vector F_spatial; Step 202: The structured data is processed using a Long Short-Term Memory (LSTM) network to obtain the temporal feature vector F_temporal; Step 203: The spatial feature vector F_spatial and the temporal feature vector F_temporal are fused together using feature concatenation or weighted averaging to obtain the fused feature vector F_fused = [F_spatial, F_temporal].

4. The method according to claim 1, characterized in that, In step 300, the model training is performed using an adaptive optimization algorithm.

5. The method according to claim 1, characterized in that, Step 600 involves preprocessing and initializing the data blocks in the fast storage or cache. Specific operations include: (1) Clean and convert the format of the extracted data blocks; (2) Establish an index structure for the preprocessed data blocks; (3) Load metadata information for data blocks with established index structures.

6. The method according to claim 5, characterized in that, Use B-trees or hash tables to build the index structure for data blocks.

7. The method according to claim 5, characterized in that, Metadata information includes access permissions and storage paths for data blocks.

8. The method according to claim 1, characterized in that, Step 600 involves tiered storage, specifically including the following operations: Based on the data block's popularity level, its distribution across different storage media is dynamically adjusted: hot data is stored in high-speed SSDs or memory; warm data is stored in medium-speed HDDs; and cold data is stored in low-speed storage media.

9. A storage optimization device based on seismic data processing, characterized in that, include: The data acquisition and parsing unit is used to collect log information through data access logs, parse the log information to convert it into structured data, and store it in a distributed manner. The feature extraction and fusion unit is used to extract the spatial and temporal features of the structured data respectively, and to fuse the spatial and temporal features to obtain the fused features; The model building unit is used to build a deep learning model using the fused features, and then train it to obtain a prediction model. The prediction unit is used to predict the data blocks, times, and frequencies to be accessed in the future based on the output of the prediction model. The data block extraction unit is used to extract the required data blocks from the distributed storage based on the prediction results and store them in fast storage or cache; Tiered storage units are used to store data blocks in fast storage or cache in tiers after preprocessing and initialization.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer-executable program, which, when executed by the computer, causes the computer to perform the steps in the storage optimization method based on seismic data processing as described in any one of claims 1-8.

Images