Data storage system and method for artificial intelligence deep learning

By introducing data acquisition, preprocessing, storage design and monitoring modules into the data storage system, it is automatically stored in different media according to the frequency and importance of data access, solving the problems of high storage costs and low access speed in the prior art, and realizing intelligent storage hierarchy and dynamic adjustment.

CN120578346APending Publication Date: 2025-09-02杭州中谦科技有限公司
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202510779740.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-12
Publication Date
2025-09-02

AI Technical Summary

Technical Problem

Existing data storage systems cannot automatically store data in different types of storage media based on the frequency and importance of access, resulting in high storage costs and low access speeds.

Method used

The data acquisition module, data preprocessing module, data storage design module, real-time data processing module, data backup module and data monitoring and maintenance module are adopted to identify data sources, automatically detect outliers, dynamic data partitioning based on access frequency and importance, select appropriate storage media, and adjust the data partitioning strategy in real time.

Benefits of technology

Optimize storage costs, improve access speed, and adjust data partitioning strategies in real time according to changes in data volume.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120578346A_ABST
    Figure CN120578346A_ABST
Patent Text Reader

Abstract

The invention discloses a data storage system and method for artificial intelligence deep learning, and relates to the technical field of data storage, and the system comprises a data obtaining module which is used for recognizing a data source and obtaining data from the data source; the data preprocessing module is used for automatically detecting and cleaning abnormal values in the data; the data storage design module is used for automatically storing the data in different types of storage media based on the access frequency and importance of the data and establishing a dynamic data partition model; the real-time data processing module is used for storing and managing real-time data streams; and the data backup module is used for backing up the data. According to the method, intelligent storage layering is provided, the data can be automatically stored in different types of storage media according to the access frequency and importance of the data, the storage cost is optimized, the access speed is increased, and a data partitioning strategy can be adjusted in real time to cope with the change of the data volume by establishing a dynamic adjustment model and adopting a dynamic data partitioning technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a data storage system and method for artificial intelligence deep learning. Background Art

[0002] The storage system refers to the system in the computer composed of various storage devices for storing programs and data, control components, and equipment and algorithms for managing information scheduling. The storage system is one of the important components of the computer. The storage system provides the ability to write and read information required for the computer to work, realizing the computer's information memory function. Modern computer systems often use a multi-level storage architecture of registers, cache, main memory, and external memory. The core of the computer storage system is the memory. The memory is an indispensable memory device in the computer used to store programs and data. The internal memory mainly stores the programs and data needed for the computer's current work, including cache memory and main memory. Currently, the main component of memory is semiconductor memory. External memory mainly has three implementation methods: magnetic memory, optical memory, and semiconductor memory. Storage media include hard disks, optical disks, tapes, and mobile storage.

[0003] With the development of science and technology, the requirements for data storage are becoming higher and higher. Existing data storage systems and methods cannot automatically store data in different types of storage media according to the access frequency and importance of the data, resulting in high storage costs and low access speeds. Summary of the Invention

[0004] In order to solve the above technical problems, a data storage system and method for deep learning of artificial intelligence and the like are provided. This technical solution solves the above problems.

[0005] In order to achieve the above objects, the technical solution adopted by the present invention is: A data storage system and method for artificial intelligence deep learning, comprising: Data acquisition module: used to identify the data source and obtain data from the data source; Data preprocessing module: The data preprocessing module is electrically connected to the data acquisition module and is used to automatically detect and clean outliers, noise and errors in the data; Data storage design module: The data storage design module is electrically connected to the data preprocessing module and the data acquisition module. The data storage design module is used to automatically store data in different types of storage media based on the access frequency and importance of the data, establish a dynamic data partition model, and adjust the data partition strategy in real time based on the dynamic data partition model; Real-time data processing module: The real-time data processing module is electrically connected to the data storage design module, the data preprocessing module and the data acquisition module. The real-time data processing module is used to process the real-time data stream using the data stream processing framework, and store and manage the real-time data stream; Data backup module: The data backup module is electrically connected to the real-time data processing module, the data storage design module, the data pre-processing module, and the data acquisition module. The data backup module is used to back up data and store the backup data in multiple geographical locations; Data monitoring and maintenance module: The data monitoring and maintenance module is electrically connected to the data backup module, real-time data processing module, data storage design module, data preprocessing module and data acquisition module. The data monitoring and maintenance module is used to monitor the performance and health status of the distributed storage system and regularly check the data quality.

[0006] Preferably, the data acquisition module specifically includes: Data source identification: Review the documentation and description of the data to identify its source, including databases, APIs, file systems, and third-party services; Data extraction: Extracting raw data from identified data sources, including crawling, downloading, and reading operations.

[0007] Preferably, the data preprocessing module specifically includes: Standardization and normalization unit: convert data into a standardized format to eliminate the dimensional effects between different features; Abnormal data detection unit: randomly select multiple subsample sets from the original data, build an isolation tree for each subsample set, calculate the path length of each data point in each isolation tree, calculate the average path length of each data point in all isolation trees, calculate the anomaly score based on the average path length, determine the anomaly score threshold, and mark the data points with anomaly scores higher than this threshold as abnormal; Automatic adjustment unit: corrects abnormal points based on detected data abnormal points.

[0008] Preferably, the data storage design module includes: Data classification and labeling unit: collects data access logs, analyzes access frequency and importance, classifies data, and labels it as hot data, warm data, and cold data; The formula for calculating access frequency is: Where, is the access frequency, is the total number of visits, is the time window; Based on the frequency of access, the importance is ranked: Hot data: data with high access frequency and high business demand score; Warm data: medium access frequency and business need score; Cold data: low access frequency and low business demand score; Storage medium selection unit: Based on the data classification results, select the corresponding storage medium: SSD for hot data, HDD for warm data, and cloud storage for cold data; Define the performance indicators of storage media, including read and write speed and capacity; Calculate storage priority and determine data storage strategy based on the priority calculation results; The storage priority calculation formula is: Where, For the The storage priority of each data item, is the weight coefficient of access frequency, No. The access frequency of each data item, The weight coefficient for scoring business requirements, For the Business requirement score for each data item; Dynamic data partitioning unit: monitors data access patterns and storage load in real time, collecting data access frequency, storage usage, and performance metrics; Dynamic adjustment unit: Based on real-time monitoring data, a dynamic adjustment model is established to adjust the partitioning strategy; Based on the calculation results of the dynamic adjustment model, the adjusted data partitioning strategy is implemented to verify the adjustment effect.

[0009] Preferably, the dynamic adjustment unit specifically includes: The dynamic adjustment model formula is: Where, is the energy consumption difference of each partition, is the current energy consumption, is the energy consumption after partitioning, is the partition capacity adjustment amount, is the partition capacity, is the total data volume, is the amount of new data, is the new partition capacity, is the capacity of the current partition, The amount of data to be migrated, is the migration ratio; Adjust the data partitioning strategy based on the new partition capacity and the amount of migrated data.

[0010] Preferably, the real-time data processing module specifically includes: Data stream receiving unit: accesses the real-time data source through the data stream processing framework, parses the received data, and converts it into the format corresponding to the data stream processing framework; Real-time data stream processing unit: performs real-time transformations on data streams and applies real-time computing operations, including aggregation, filtering, and window calculations, to merge and process data streams from different sources to obtain the final result; Data storage unit: writes processed data into the storage system in real time, including distributed databases, data warehouses, and message queues; Result output unit: supports query operations on real-time data, provides immediate analysis results, and generates real-time reports and visualization results.

[0011] Preferably, the data backup module specifically includes: Backup data determination unit: select the data and files to be backed up; Backup strategy formulation unit: determines the backup frequency, backup type, and retention period; Configure geographic location unit: select and configure multiple geographic locations for storing backup data; Execute data backup unit: Perform backup according to the policy and back up data in batches or in full to multiple designated locations.

[0012] Preferably, the data monitoring and maintenance module specifically includes: Performance monitoring unit: defines storage utilization, read / write speed, and latency, and monitors these indicators in real time; Health status check unit: manually scans storage devices to detect hardware failures and regularly generates device health status reports; Data quality check unit: regularly verify data integrity; Logging and Analysis Unit: Records logs of all monitoring and maintenance activities, including errors, warnings, and system status.

[0013] Data storage methods for artificial intelligence deep learning, including: Identify the data source and obtain data from it; Automatically detect and clean outliers, noise, and errors in data; Automatically store data in different types of storage media based on data access frequency and importance, and use dynamic data partitioning technology to adjust data partitioning strategies in real time; Use data stream processing framework to process real-time data streams, store and manage real-time data streams; Back up data and store it in multiple geographical locations; Monitor the performance and health of storage systems and regularly check data quality.

[0014] Preferably, data is automatically stored in different types of storage media based on the access frequency and importance of the data, and dynamic data partitioning technology is used to adjust the data partitioning strategy in real time, including: Collect data access logs, analyze access frequency and importance, classify data, and mark it as hot data, warm data, and cold data; Based on the frequency of access, the importance is ranked; Based on the data classification results, select the corresponding storage medium; Calculate storage priority and determine data storage strategy based on the priority calculation results; Based on the calculation results of the dynamic adjustment model, the adjusted data partitioning strategy is implemented to verify the adjustment effect.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes intelligent storage tiering, which can automatically store data in different types of storage media according to the access frequency and importance of the data, optimize storage costs, and improve access speed. By establishing a dynamic adjustment model and adopting dynamic data partitioning technology, the data partitioning strategy can be adjusted in real time to cope with changes in data volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a system framework diagram of the present invention; Figure 2 This is a diagram of the internal system framework of the data acquisition module in the present invention; Figure 3 This is a diagram of the internal system framework of the data preprocessing module in the present invention; Figure 4 This is an internal system framework diagram of the data storage design module in the present invention; Figure 5 This is a diagram of the internal system framework of the real-time data processing module in the present invention; Figure 6 This is a diagram of the internal system framework of the data backup module in the present invention; Figure 7 This is the internal system framework diagram of the data monitoring and maintenance module in the present invention; Figure 8 is a flow chart of the method of the present invention; Figure 9 This is a specific process framework diagram for the data storage design in the present invention. DETAILED DESCRIPTION

[0017] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0018] Reference Figure 1 As shown, a data storage system and method for artificial intelligence deep learning include: Data acquisition module: used to identify the data source and obtain data from the data source; Data preprocessing module: The data preprocessing module is electrically connected to the data acquisition module and is used to automatically detect and clean outliers, noise and errors in the data; Data storage design module: The data storage design module is electrically connected to the data preprocessing module and the data acquisition module. The data storage design module is used to automatically store data in different types of storage media based on the access frequency and importance of the data, establish a dynamic data partition model, and adjust the data partition strategy in real time based on the dynamic data partition model; Real-time data processing module: The real-time data processing module is electrically connected to the data storage design module, the data preprocessing module and the data acquisition module. The real-time data processing module is used to process the real-time data stream using the data stream processing framework, and store and manage the real-time data stream; Data backup module: The data backup module is electrically connected to the real-time data processing module, the data storage design module, the data pre-processing module, and the data acquisition module. The data backup module is used to back up data and store the backup data in multiple geographical locations; Data monitoring and maintenance module: The data monitoring and maintenance module is electrically connected to the data backup module, real-time data processing module, data storage design module, data preprocessing module and data acquisition module. The data monitoring and maintenance module is used to monitor the performance and health status of the distributed storage system and regularly check the data quality.

[0019] Reference Figure 2 As shown, the data acquisition module specifically includes: Data source identification: Review the documentation and description of the data to identify its source, including databases, APIs, file systems, and third-party services; Data extraction: Extracting raw data from identified data sources, including crawling, downloading, and reading operations.

[0020] Reference Figure 3 As shown, the data preprocessing module specifically includes: Standardization and normalization unit: convert data into a standardized format to eliminate the dimensional effects between different features; Abnormal data detection unit: randomly select multiple subsample sets from the original data, build an isolation tree for each subsample set, calculate the path length of each data point in each isolation tree, calculate the average path length of each data point in all isolation trees, calculate the anomaly score based on the average path length, determine the anomaly score threshold, and mark the data points with anomaly scores higher than this threshold as abnormal; Automatic adjustment unit: corrects abnormal points based on detected data abnormal points.

[0021] Reference Figure 4 As shown, the data storage design module includes: Data classification and labeling unit: collects data access logs, analyzes access frequency and importance, classifies data, and labels it as hot data, warm data, and cold data; The formula for calculating access frequency is: Where, is the access frequency, is the total number of visits, is the time window; Based on the frequency of access, the importance is ranked: Hot data: data with high access frequency and high business demand score; Warm data: medium access frequency and business need score; Cold data: low access frequency and low business demand score; Storage medium selection unit: Based on the data classification results, select the corresponding storage medium: SSD for hot data, HDD for warm data, and cloud storage for cold data; Define the performance indicators of storage media, including read and write speed and capacity; Calculate storage priority and determine data storage strategy based on the priority calculation results; The storage priority calculation formula is: Where, For the The storage priority of each data item, is the weight coefficient of access frequency, No. The access frequency of each data item, The weight coefficient for scoring business requirements, For the Business requirement score for each data item; Dynamic data partitioning unit: monitors data access patterns and storage load in real time, collecting data access frequency, storage usage, and performance metrics; Dynamic adjustment unit: Based on real-time monitoring data, a dynamic adjustment model is established to adjust the partitioning strategy; Based on the calculation results of the dynamic adjustment model, the adjusted data partitioning strategy is implemented to verify the adjustment effect.

[0022] Reference Figure 5 As shown, the dynamic adjustment unit specifically includes: The dynamic adjustment model formula is: In the formula, in the formula, is the energy consumption difference of each partition, is the current energy consumption, is the energy consumption after partitioning, is the partition capacity adjustment amount, is the partition capacity, is the total data volume, is the amount of new data, is the new partition capacity, is the capacity of the current partition, The amount of data to be migrated, is the migration ratio; Adjust the data partitioning strategy based on the new partition capacity and the amount of migrated data.

[0023] Reference Figure 6 As shown, the real-time data processing module specifically includes: Data stream receiving unit: accesses the real-time data source through the data stream processing framework, parses the received data, and converts it into the format corresponding to the data stream processing framework; Real-time data stream processing unit: performs real-time transformations on data streams and applies real-time computing operations, including aggregation, filtering, and window calculations, to merge and process data streams from different sources to obtain the final result; Data storage unit: writes processed data into the storage system in real time, including distributed databases, data warehouses, and message queues; Result output unit: supports query operations on real-time data, provides immediate analysis results, and generates real-time reports and visualization results.

[0024] Preferably, the data backup module specifically includes: Backup data determination unit: select the data and files to be backed up; Backup strategy formulation unit: determines the backup frequency, backup type, and retention period; Configure geographic location unit: select and configure multiple geographic locations for storing backup data; Execute data backup unit: Perform backup according to the policy and back up data in batches or in full to multiple designated locations.

[0025] Reference Figure 7 As shown, the data monitoring and maintenance module specifically includes: Performance monitoring unit: defines storage utilization, read / write speed, and latency, and monitors these indicators in real time; Health status check unit: manually scans storage devices to detect hardware failures and regularly generates device health status reports; Data quality check unit: regularly verify data integrity; Logging and Analysis Unit: Records logs of all monitoring and maintenance activities, including errors, warnings, and system status.

[0026] Reference Figure 8 As shown, a data storage method for artificial intelligence deep learning includes: Identify the data source and obtain data from it; Automatically detect and clean outliers, noise, and errors in data; Automatically store data in different types of storage media based on data access frequency and importance, and use dynamic data partitioning technology to adjust data partitioning strategies in real time; Use data stream processing framework to process real-time data streams, store and manage real-time data streams; Back up data and store it in multiple geographical locations; Monitor the performance and health of storage systems and regularly check data quality.

[0027] Reference Figure 9 As shown, data is automatically stored in different types of storage media based on the access frequency and importance of the data. Dynamic data partitioning technology is used to adjust the data partitioning strategy in real time, including: Collect data access logs, analyze access frequency and importance, classify data, and mark it as hot data, warm data, and cold data; Based on the frequency of access, the importance is ranked; Based on the data classification results, select the corresponding storage medium; Calculate storage priority and determine data storage strategy based on the priority calculation results; Based on the calculation results of the dynamic adjustment model, the adjusted data partitioning strategy is implemented to verify the adjustment effect.

[0028] The use process of the present invention is: Step 1: Review the documentation and description of the data and identify its sources, including databases, APIs, file systems, and third-party services. Step 2: Extracting raw data from the identified data source, including crawling, downloading, and reading operations; Step 3: Convert the data into a standardized format to eliminate the dimensional effects between different features; Step 4: Randomly select multiple subsample sets from the original data. For each subsample set, build an isolation tree. For each data point, calculate its path length in each isolation tree. Calculate the average path length of each data point in all isolation trees. Calculate the anomaly score based on the average path length. Determine the anomaly score threshold and mark the data points with anomaly scores higher than this threshold as anomalies. Step 5: Automatic adjustment unit: Automatically adjust the preprocessing strategy according to the characteristics of the data and the preprocessing results; Step 6: Automation and Continuous Monitoring Unit: Establish an automated data pre-processing process and regularly monitor data quality; Step 7: Collect data access logs, analyze access frequency and importance, and classify the data into hot data, warm data, and cold data. Step 8: Based on the data classification results, select the corresponding storage medium: SSD for hot data, HDD for warm data, and cloud storage for cold data; Step 9: Define the performance indicators of the storage media, including read and write speed and capacity; Step 10: Calculate the storage priority and determine the data storage strategy based on the priority calculation result; Step 11: Based on real-time monitoring data, a dynamic adjustment model is established to adjust the partitioning strategy; Step 12: Based on the calculation results of the dynamic adjustment model, implement the adjusted data partitioning strategy and verify the adjustment effect; Step 13: Access the real-time data source through the data stream processing framework, parse the received data, and convert it into the format corresponding to the data stream processing framework; Step 14: Perform real-time transformations on the data streams and apply real-time computing operations, including aggregation, filtering, and window calculations, to merge and process data streams from different sources to obtain the final result. Step 15: Write the processed data to the storage system in real time, including distributed databases, data warehouses, and message queues; Step 16: Support query operations on real-time data, provide instant analysis results, and generate real-time reports and visualization results; Step 17: Define storage utilization, read / write speeds, and latency, and monitor these metrics in real time. Step 18: Manually scan storage devices to detect hardware failures and generate device health status reports regularly. Step 19: Data Quality Check Unit: Regularly verify data integrity; Step 20: Record a log of all monitoring and maintenance activities, including errors, warnings, and system status.

[0029] To sum up, the advantages of the present invention are: the present invention proposes intelligent storage tiering, which can automatically store data in different types of storage media according to the access frequency and importance of the data, optimize the storage cost, and improve the access speed. By establishing a dynamic adjustment model and adopting dynamic data partitioning technology, the data partitioning strategy can be adjusted in real time to cope with changes in data volume.

[0030] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A data storage system for artificial intelligence deep learning, characterized in that: The data storage system includes: Data acquisition module: used to identify the data source and obtain data from the data source; Data preprocessing module: The data preprocessing module is electrically connected to the data acquisition module and is used to automatically detect and clean outliers, noise and errors in the data; Data storage design module: The data storage design module is electrically connected to the data preprocessing module and the data acquisition module. The data storage design module is used to automatically store data in different types of storage media based on the access frequency and importance of the data, establish a dynamic data partition model, and adjust the data partition strategy in real time based on the dynamic data partition model; Real-time data processing module: The real-time data processing module is electrically connected to the data storage design module, the data preprocessing module and the data acquisition module. The real-time data processing module is used to process the real-time data stream using the data stream processing framework, and store and manage the real-time data stream; Data backup module: The data backup module is electrically connected to the real-time data processing module, the data storage design module, the data preprocessing module and the data acquisition module, and is used to back up data; Data monitoring and maintenance module: The data monitoring and maintenance module is electrically connected to the data backup module, real-time data processing module, data storage design module, data preprocessing module and data acquisition module. The data monitoring and maintenance module is used to monitor the performance and health status of the distributed storage system and regularly check the data quality.

2. The data storage system for artificial intelligence deep learning according to claim 1, characterized in that: The data acquisition module specifically includes: Data source identification: Review the documentation and description of the data to identify its source, including databases, APIs, file systems, and third-party services; Data extraction: Extracting raw data from identified data sources, including crawling, downloading, and reading operations.

3. The data storage system for artificial intelligence deep learning according to claim 2, characterized in that: The data preprocessing module specifically includes: Standardization and normalization unit: convert data into a standardized format to eliminate the dimensional effects between different features; Abnormal data detection unit: randomly select multiple subsample sets from the original data, build an isolation tree for each subsample set, calculate the path length of each data point in each isolation tree, calculate the average path length of each data point in all isolation trees, calculate the anomaly score based on the average path length, determine the anomaly score threshold, and mark the data points with anomaly scores higher than this threshold as abnormal; Automatic adjustment unit: corrects abnormal points based on detected data abnormal points.

4. The data storage system for artificial intelligence deep learning according to claim 3, characterized in that: The data storage design module includes: Data classification and labeling unit: collects data access logs, analyzes access frequency and importance, classifies data, and labels it as hot data, warm data, and cold data; The formula for calculating access frequency is: Where, is the access frequency, is the total number of visits, is the time window; Based on the frequency of access, the importance is ranked: Hot data: data with high access frequency and high business demand score; Warm data: medium access frequency and business need score; Cold data: low access frequency and low business demand score; Storage medium selection unit: Based on the data classification results, select the corresponding storage medium: SSD for hot data, HDD for warm data, and cloud storage for cold data; Define the performance indicators of storage media, including read and write speed and capacity; Calculate storage priority and determine data storage strategy based on the priority calculation results; The storage priority calculation formula is: Where, For the The storage priority of each data item, is the weight coefficient of access frequency, No. The access frequency of each data item, The weight coefficient for scoring business requirements, For the Business requirement score for each data item; Dynamic data partitioning unit: real-time monitoring of data partitioning; Dynamic adjustment unit: Based on real-time monitoring data, a dynamic adjustment model is established to adjust the partitioning strategy; Based on the calculation results of the dynamic adjustment model, the adjusted data partitioning strategy is implemented to verify the adjustment effect.

5. The data storage system for artificial intelligence deep learning according to claim 4, characterized in that: The dynamic adjustment model is as follows: Where, is the energy consumption difference of each partition, is the current energy consumption, is the energy consumption after partitioning, is the partition capacity adjustment amount, is the partition capacity, is the total data volume, is the amount of new data, is the new partition capacity, is the capacity of the current partition, The amount of data to be migrated, is the migration ratio; Adjust the data partitioning strategy based on the new partition capacity and the amount of migrated data.

6. The data storage system for artificial intelligence deep learning according to claim 5, characterized in that: The real-time data processing module specifically includes: Data stream receiving unit: accesses the real-time data source through the data stream processing framework, parses the received data, and converts it into the format corresponding to the data stream processing framework; Real-time data stream processing unit: performs real-time transformations on data streams and applies real-time computing operations, including aggregation, filtering, and window calculations, to merge and process data streams from different sources to obtain the final result; Data storage unit: writes processed data into the storage system in real time, including distributed databases, data warehouses, and message queues; Result output unit: supports query operations on real-time data, provides immediate analysis results, and generates real-time reports and visualization results.

7. The data storage system for artificial intelligence deep learning according to claim 6, characterized in that: The data backup module specifically includes: Backup data determination unit: select the data and files to be backed up; Backup strategy formulation unit: determines the backup frequency, backup type, and retention period; Configure geographic location unit: select and configure multiple geographic locations for storing backup data; Execute data backup unit: Perform backup according to the policy and back up data in batches or in full to multiple designated locations.

8. The data storage system for artificial intelligence deep learning according to claim 7, characterized in that: The data monitoring and maintenance module specifically includes: Performance monitoring unit: defines storage utilization, read / write speed, and latency, and monitors these indicators in real time; Health status check unit: manually scans storage devices to detect hardware failures and regularly generates device health status reports; Data quality check unit: regularly verify data integrity; Logging and Analysis Unit: Records logs of all monitoring and maintenance activities, including errors, warnings, and system status.

9. A data storage method for artificial intelligence deep learning, characterized in that: Data storage methods include: Identify the data source and obtain data from it; Automatically detect and clean outliers, noise, and errors in data; Automatically store data in different types of storage media based on data access frequency and importance, and use dynamic data partitioning technology to adjust data partitioning strategies in real time; Use data stream processing framework to process real-time data streams, store and manage real-time data streams; Back up data and store it in multiple geographical locations; Monitor the performance and health of storage systems and regularly check data quality.

10. The data storage method for artificial intelligence deep learning according to claim 9, characterized in that: Automatically store data in different types of storage media based on data access frequency and importance, using dynamic data partitioning technology to adjust data partitioning strategies in real time, including: Collect data access logs, analyze access frequency and importance, classify data, and mark it as hot data, warm data, and cold data; Based on the frequency of access, the importance is graded; Based on the data classification results, select the corresponding storage medium; Calculate storage priority and determine data storage strategy based on the priority calculation results; Based on the calculation results of the dynamic adjustment model, the adjusted data partitioning strategy is implemented to verify the adjustment effect.

Citation Information

Cited By

  • Open source defense information data connection and reorganization processing method and system

    CN121350093A

  • Big data stream processing method and system based on machine learning

    CN121957647A