Computer data storage system

Abstract

The invention discloses a computer data storage system which comprises a computer network data interface, a data acquisition module, a data control module, a network data analysis module, a data compression module, a data storage module and a data access optimization module. Wherein the computer network data interface is responsible for receiving and sending data; the data acquisition module and the data control module work cooperatively to perform primary processing on the received data; the network data analysis module integrates a machine learning algorithm and is used for data feature extraction, intelligent classification and useless data identification and elimination; the method has the advantages that through intelligent classification and compression, data redundancy is effectively reduced, and the effective utilization rate of storage space is increased; a data indexing and caching mechanism is optimized, so that the data access speed is remarkably increased, and the requirements of quick query and analysis are met; distributed processing of data is supported, expansion is easy, and data storage requirements of different scales are met.

Description

Technical Field

[0001] The invention belongs to the technical field of data storage, and in particular relates to a computer data storage system. Background Art

[0002] A storage system refers to a system in a computer that is composed of various storage devices for storing programs and data, control components, and equipment (hardware) and algorithms (software) for managing information scheduling. The main memory of a computer cannot simultaneously meet the requirements of fast access speed, large storage capacity, and low cost. There must be a multi-level hierarchical memory with speeds from slow to fast and capacities from large to small in the computer to form a storage system with acceptable performance with the optimal control scheduling algorithm and reasonable cost. The storage hierarchy in a computer system can be divided into three levels: cache memory, main memory, and auxiliary memory. Cache memory is used to improve the speed matching problem between the main memory and the central processing unit. Auxiliary memory is used to expand storage space. During information access, the storage system must complete the conversion between the logical address space and the physical address space, and reasonably manage the storage system resources.

[0003] With the rapid development of information technology, the demand for computer data storage is growing, and the performance requirements for storage systems are also increasing. Although traditional storage systems have built a multi-level storage architecture to balance speed, capacity and cost, they still face problems such as high data redundancy, low storage efficiency and slow access speed. Especially in the big data environment, how to effectively manage massive data and achieve rapid data access and analysis has become a technical problem that needs to be solved urgently.

[0004] The computer data storage system with publication number CN107463340A discloses a computer network data interface, wherein the signal output end of the computer network interface is connected to a data acquisition module and a data control module, and the signal output ends of the data acquisition module and the data control module are connected to a network data analysis module, wherein the network data analysis module divides data into useful data and useless data, wherein the useless data is directly deleted, and the useful data is divided into regular data and redundant data, wherein the regular data is directly stored, and the redundant data is stored after data compression; the computer network data interface is used for port access to data under a network computer model; the data acquisition module and the data control module are used for collecting and controlling received data; and the network data analysis module is used for analyzing, classifying and detecting received data. The patent has deficiencies in data feature extraction, intelligent classification, and rapid access and screening of data. Summary of the invention

[0005] The purpose of the present invention is to provide a computer data storage system that can efficiently manage data, reduce redundancy, and accelerate data access and screening, thereby meeting the urgent needs for data storage and processing in the big data era.

[0006] To achieve the above object, the present invention provides the following technical solutions: a computer data storage system, including a computer network data interface, a data acquisition module, a data control module, a network data analysis module, a data compression module, a data storage module and a data access optimization module; wherein,

[0007] The computer network data interface is responsible for receiving and sending data;

[0008] The data acquisition module works in conjunction with the data control module to perform preliminary processing on the received data;

[0009] The network data analysis module integrates machine learning algorithms for data feature extraction, intelligent classification, and identification and elimination of useless data;

[0010] The data compression module uses a compression algorithm to compress redundant data;

[0011] The data storage module is responsible for storing processed data and supports fast indexing and retrieval;

[0012] The data access optimization module implements fast fixed-point access and screening of data by building data index and cache mechanism.

[0013] As a preferred technical solution of the present invention, the data acquisition module and the data control module work together to perform preliminary processing on the received data, and the implementation method is as follows:

[0014] In the data acquisition module, one or more data receiving points are usually set up to obtain data from the computer network data interface. When the data arrives, the data acquisition module will capture it and store it in a temporary buffer;

[0015] The data control module is responsible for managing and scheduling the processing flow of these data, and deciding how to process the data based on the type, source or other attributes of the data;

[0016] An effective communication mechanism is established between the data acquisition module and the data control module; after capturing the data, the data acquisition module sends a notification to the data control module or directly passes the data to the control module, and the control module decides the next processing method based on preset rules or strategies.

[0017] As a preferred technical solution of the present invention, the receiving point includes a file reader and a network socket listener.

[0018] As a preferred technical solution of the present invention, an effective communication mechanism is established between the data acquisition module and the data control module, which is realized through shared memory, message queue and database.

[0019] As a preferred technical solution of the present invention, the network data analysis module integrates a machine learning algorithm for data feature extraction, intelligent classification, and identification and elimination of useless data. The implementation method is as follows:

[0020] The network data analysis module integrates a variety of machine learning algorithms, which are used to analyze and mine data to extract useful features and patterns;

[0021] The network data analysis module first preprocesses the data and then applies a specific machine learning algorithm to extract feature vectors of the data; these feature vectors represent certain key attributes or patterns of the data;

[0022] Intelligent classification is achieved by training a classifier, which divides the data into different categories based on its feature vector;

[0023] The identification and elimination of useless data is achieved by building an evaluation model that evaluates the usefulness of data based on the feature vector of the data. When the data is evaluated as useless or redundant, it will be eliminated.

[0024] As a preferred technical solution of the present invention, the machine learning algorithm includes a clustering algorithm, a classification algorithm, and a regression algorithm.

[0025] As a preferred technical solution of the present invention, the data compression module uses a compression algorithm to compress the redundant data, and the implementation method is as follows:

[0026] The data compression module first analyzes the type and characteristics of the data, and then selects a compression algorithm for compression processing; during the compression process, the module removes redundant information in the data while maintaining the readability and decompressibility of the data.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] Through intelligent classification and compression, data redundancy is effectively reduced and the effective utilization of storage space is increased;

[0029] Optimize data indexing and caching mechanisms to significantly improve data access speed and meet the needs of fast query and analysis;

[0030] Supports distributed data processing, is easy to expand, and adapts to data storage needs of different scales;

[0031] Automated data management and optimization mechanisms reduce manual intervention and lower operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the computer data storage system of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example 1

[0035] See also Figure 1 , is the first embodiment of the present invention, which provides a computer data storage system, including a computer network data interface, a data acquisition module, a data control module, a network data analysis module, a data compression module, a data storage module and a data access optimization module; wherein,

[0036] The computer network data interface is responsible for receiving and sending data;

[0037] The data acquisition module works in conjunction with the data control module to perform preliminary processing on the received data;

[0038] The network data analysis module integrates machine learning algorithms for data feature extraction, intelligent classification, and identification and elimination of useless data;

[0039] The data compression module uses a compression algorithm to compress redundant data;

[0040] The data storage module is responsible for storing processed data and supports fast indexing and retrieval;

[0041] The data access optimization module implements fast fixed-point access and screening of data by building data index and cache mechanism.

[0042] In this embodiment, preferably, the data acquisition module and the data control module work together to perform preliminary processing on the received data, and the implementation method is as follows:

[0043] In the data acquisition module, one or more data receiving points are usually set up to obtain data from the computer network data interface or other data sources. These receiving points include file readers, network socket listeners, etc. When the data arrives, the data acquisition module will capture it and store it in a temporary buffer;

[0044] The data control module is responsible for managing and scheduling the processing flow of these data, and deciding how to process the data based on the type, source or other attributes of the data;

[0045] In order to achieve collaborative work, an effective communication mechanism needs to be established between the data acquisition module and the data control module, which can be achieved through shared memory, message queues, and databases; after capturing the data, the data acquisition module will send a notification to the data control module or directly pass the data to the control module; the control module decides the next processing method based on preset rules or strategies.

[0046] In this embodiment, preferably, the network data analysis module integrates a machine learning algorithm for data feature extraction, intelligent classification, and identification and elimination of useless data, and the implementation method is as follows:

[0047] Network data analysis modules usually integrate multiple machine learning algorithms, such as clustering algorithms, classification algorithms, regression algorithms, etc. These algorithms are used to analyze and mine data to extract useful features and patterns;

[0048] In order to achieve data feature extraction, the network data analysis module will first pre-process the data, such as denoising, standardization, etc. Then, it will apply specific machine learning algorithms to extract feature vectors of the data, which can represent certain key attributes or patterns of the data;

[0049] Intelligent classification is achieved by training a classifier, which can classify data into different categories based on its feature vectors. This usually involves a training process in which the classifier learns how to correctly classify data of known categories.

[0050] The identification and elimination of useless data is achieved by building an evaluation model that can evaluate the usefulness of data based on the feature vector of the data. If the data is evaluated as useless or redundant, it will be eliminated.

[0051] In this embodiment, preferably, the data compression module uses a compression algorithm to compress the redundant data, and the implementation method is as follows:

[0052] The data compression module usually uses compression algorithms to reduce the storage space of data. These algorithms can be divided into lossless compression and lossy compression;

[0053] Lossless compression algorithms attempt to reduce redundant information in data without losing any of the original data. These algorithms are usually used in scenarios where data integrity needs to be maintained.

[0054] Lossy compression algorithms sacrifice some data quality in exchange for a higher compression ratio. These algorithms are usually used to compress media data such as images and audio.

[0055] The data compression module first analyzes the type and characteristics of the data, and then selects the most appropriate compression algorithm for compression processing. During the compression process, the module will try to remove redundant information in the data while maintaining the readability and decompressibility of the data.

[0056] Example 2

[0057] See also Figure 1 , which is the second embodiment of the present invention, is based on the previous embodiment, except that:

[0058] The data storage module is responsible for storing processed data and supports fast indexing and retrieval. The implementation method is as follows:

[0059] The data storage module usually uses a database system or a file system to store data. These systems provide efficient data storage and retrieval functions.

[0060] In order to achieve fast indexing and retrieval, the data storage module will use various indexing techniques; these indexes can speed up the data search and retrieval process;

[0061] In addition, the data storage module can also use partitioning, sharding and other technologies to optimize data storage and access performance; by dividing the data into smaller blocks or partitions, the amount of data that needs to be scanned for each query can be reduced, thereby improving query efficiency.

[0062] The data access optimization module implements fast fixed-point access and filtering of data by building data index and cache mechanism. The implementation method is as follows:

[0063] The data access optimization module is mainly responsible for improving the efficiency and speed of data access. To achieve this goal, various data indexes and cache mechanisms are constructed. Data index is a data structure used to speed up the data search and retrieval process. The data access optimization module selects the appropriate index type based on the data type and query requirements. For example, for frequently executed range queries, B-tree or R-tree indexes are selected; for exact match queries, hash table indexes are selected.

[0064] The cache mechanism is a technology that stores frequently used data or query results in memory to improve access speed. The data access optimization module uses the cache to store frequently accessed data or query results. When the same query is executed again, the result can be obtained directly from the cache without accessing the underlying database or file system again.

[0065] The data access optimization module can also use other technologies to optimize data access performance, such as query rewriting, pre-computation, etc. These technologies can further reduce the complexity and execution time of queries, thereby improving the response speed and performance of the entire system.

[0066] Experimental verification:

[0067]

[0068] It can be seen from the table that the computer data storage system of the present invention is superior to the patent CN107463340A cited in the background technology in terms of data classification accuracy, data redundancy, data access speed, storage space utilization, distributed processing capability, degree of automation and operation and maintenance cost; these improvements enable the present invention to better meet the needs of data storage and processing in the big data era and improve the efficiency and reliability of the data storage system.

[0069] Although the embodiments of the present invention have been shown and described, as detailed above, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A computer data storage system, characterized in that: It includes a computer network data interface, a data acquisition module, a data control module, a network data analysis module, a data compression module, a data storage module and a data access optimization module; among which, The computer network data interface is responsible for receiving and sending data; The data acquisition module works in conjunction with the data control module to perform preliminary processing on the received data; The network data analysis module integrates machine learning algorithms for data feature extraction, intelligent classification, and identification and elimination of useless data; The data compression module uses a compression algorithm to compress redundant data; The data storage module is responsible for storing processed data and supports fast indexing and retrieval; The data access optimization module implements fast fixed-point access and screening of data by building data index and cache mechanism.

2. A computer data storage system according to claim 1, characterized in that: The data acquisition module works with the data control module to perform preliminary processing on the received data. The implementation method is as follows: In the data acquisition module, one or more data receiving points are usually set up to obtain data from the computer network data interface. When the data arrives, the data acquisition module will capture it and store it in a temporary buffer; The data control module is responsible for managing and scheduling the processing flow of these data, and deciding how to process the data based on the type, source or other attributes of the data; An effective communication mechanism is established between the data acquisition module and the data control module; after capturing the data, the data acquisition module sends a notification to the data control module or directly passes the data to the control module, and the control module decides the next processing method based on preset rules or strategies.

3. A computer data storage system according to claim 2, characterized in that: The receiving point includes a file reader and a network socket listener.

4. A computer data storage system according to claim 2, characterized in that: An effective communication mechanism is established between the data acquisition module and the data control module, which is realized through shared memory, message queue and database.

5. A computer data storage system according to claim 1, characterized in that: The network data analysis module integrates machine learning algorithms for data feature extraction, intelligent classification, and identification and elimination of useless data. The implementation method is as follows: The network data analysis module integrates a variety of machine learning algorithms, which are used to analyze and mine data to extract useful features and patterns; The network data analysis module first preprocesses the data, and then applies a specific machine learning algorithm to extract the feature vector of the data; These feature vectors represent some key properties or patterns of the data; Intelligent classification is achieved by training a classifier, which divides the data into different categories based on its feature vector; The identification and elimination of useless data is achieved by building an evaluation model that evaluates the usefulness of data based on the feature vector of the data. When the data is evaluated as useless or redundant, it will be eliminated.

6. A computer data storage system according to claim 5, characterized in that: The machine learning algorithms include clustering algorithms, classification algorithms, and regression algorithms.

7. A computer data storage system according to claim 1, characterized in that: The data compression module uses a compression algorithm to compress redundant data. The implementation method is as follows: The data compression module first analyzes the type and characteristics of the data, and then selects a compression algorithm for compression processing; during the compression process, the module removes redundant information in the data while maintaining the readability and decompressibility of the data.

8. A computer data storage system according to claim 7, characterized in that: The compression algorithms include lossless compression and lossy compression.

9. A computer data storage system according to claim 1, characterized in that: The data storage module is responsible for storing processed data and supports fast indexing and retrieval. The implementation method is as follows: The data storage module uses database systems or file systems to store data, which provide efficient data storage and retrieval functions; The data storage module uses indexing technology to accelerate the data search and retrieval process.

10. A computer data storage system according to claim 9, characterized in that: The data storage module can also use partitioning and sharding technology to optimize data storage and access performance.

Images