Computer equipment based on big data processing

Abstract

The invention discloses a computer device based on big data processing, comprising: a computing node comprising at least one central processing unit (CPU); a storage node comprising at least one solid state disk (SSD); the intelligent distributed storage system is used for dynamically adjusting the position of the data according to the data access mode; the self-adaptive load balancing framework is used for dynamically adjusting distribution of computing resources according to real-time system load; the high-speed network is used for connecting the computing nodes and the storage nodes, data access delay is remarkably reduced through an intelligent data distribution strategy, reliability and data security of the system are improved through data redundancy and replication, required storage space is reduced through an efficient data compression algorithm, storage cost is reduced, and the data storage efficiency is improved. By dynamically adjusting and computing resource allocation, the resource utilization rate is improved, the task response time is shortened, efficient task processing is ensured through intelligent task scheduling, and the service quality and the system reliability are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a computer device based on big data processing. Background Art

[0002] With the rapid development of the Internet and the Internet of Things (IoT), the speed and volume of data generated are increasing exponentially, placing higher demands on big data processing technologies. Traditional data processing systems face challenges such as slow processing speeds and low storage efficiency, making it difficult to meet the growing demand for data processing. In particular, in sectors such as finance, healthcare, and e-commerce, the ability to efficiently process massive amounts of data is crucial to business success.

[0003] In existing technologies, big data processing systems mainly face the following challenges: Processing speed: As the amount of data increases, traditional data processing methods cannot meet the needs of real-time processing, resulting in increased processing delays. Storage efficiency: Large amounts of data storage not only take up expensive storage space, but also increase the cost of data management and maintenance. Computing resource allocation: When processing big data, how to reasonably allocate computing resources to maximize processing efficiency is an important issue. Data security: As the value of data increases, protecting data from attacks and unauthorized access becomes increasingly important.

[0004] To address these problems, researchers have developed a variety of solutions, such as distributed computing frameworks (such as Apache Hadoop and Apache Spark), cloud computing platforms, and parallel processing technologies. Although these technologies have improved the efficiency of data processing to a certain extent, they still have limitations when faced with extremely large data sets. Although existing technologies have made certain progress, there are still some technical challenges, such as: Intelligent optimization of data distribution: How to intelligently distribute data according to data access patterns to reduce data access latency. Dynamic scheduling of computing resources: How to dynamically adjust the allocation of computing resources according to real-time load conditions to improve resource utilization. Data security and privacy protection: How to protect data security and privacy while still being able to process data efficiently.

[0005] Therefore, a computer device based on big data processing is proposed. Summary of the Invention

[0006] The present invention aims to address shortcomings in existing technologies, including intelligent optimization of data distribution: how to intelligently distribute data based on data access patterns to reduce data access latency; dynamic scheduling of computing resources: how to dynamically adjust the allocation of computing resources based on real-time load conditions to improve resource utilization; and data security and privacy protection: how to efficiently process data while protecting data security and privacy. This invention proposes a computer device based on big data processing.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A computer device based on big data processing, comprising:

[0009] at least one computing node, the computing node comprising at least one high-performance central processing unit (CPU);

[0010] at least one storage node, wherein the storage node comprises at least one high-speed solid-state drive (SSD);

[0011] An intelligent distributed storage system that dynamically adjusts data relocation based on data access patterns to reduce read and write latency;

[0012] An adaptive load balancing framework that dynamically adjusts the allocation of computing resources based on real-time system load;

[0013] A high-speed network is used to connect the computing node and the storage node.

[0014] The above technical solution further includes:

[0015] Furthermore, the intelligent distributed storage system includes: a data redundancy and replication module for determining the number of copies of data based on the importance and access frequency of the data; and a data compression and encryption module for reducing storage space requirements and protecting data security.

[0016] Furthermore, the adaptive load balancing framework includes: a load monitoring module for real-time monitoring of the CPU usage, memory usage and other resource usage of the computing node; a resource scheduling module for dynamically adjusting the allocation of computing resources according to the real-time load conditions; and a task scheduling module for prioritizing tasks according to their attributes and allocating tasks to the most suitable computing nodes.

[0017] Furthermore, the intelligent distributed storage system includes: a data access pattern analysis module, which is used to predict future data access patterns through a machine learning algorithm; and a dynamic data migration module, which is used to migrate hot data to the SSD and cold data to the hard disk drive (HDD) based on the results of the data access pattern analysis module.

[0018] Furthermore, the adaptive load balancing framework includes: an elastic scaling module for automatically adding computing nodes when the load increases and releasing computing resources when the load decreases.

[0019] Furthermore, the computing node includes at least one graphics processing unit (GPU) and / or at least one field programmable gate array (FPGA) for accelerating specific types of data processing tasks.

[0020] Furthermore, the storage node also includes an object storage service for storing large amounts of data at low cost.

[0021] Furthermore, a method for processing big data based on a computer device for big data processing comprises the following steps:

[0022] Step A: Using the intelligent distributed storage system to dynamically adjust the location of data based on data access patterns to reduce read and write latency;

[0023] Step B: using the adaptive load balancing framework to dynamically adjust the allocation of computing resources according to the real-time system load;

[0024] Step C: executing the big data processing task through the computing node;

[0025] Step D: Storing the processing result through the storage node.

[0026] Furthermore, step A also includes: using a data access pattern analysis module to predict future data access patterns; using a dynamic data migration module to migrate hot data to SSD or memory, and migrate cold data to HDD or object storage based on the prediction results.

[0027] Furthermore, step B also includes: using a load monitoring module to monitor the CPU usage, memory usage and other resource usage of the computing node in real time; using a resource scheduling module to dynamically adjust the allocation of computing resources according to the real-time load situation; using a task scheduling module to prioritize tasks according to their attributes and assign tasks to the most suitable computing nodes.

[0028] Further, hardware architecture:

[0029] Compute nodes: These contain computing resources such as high-performance CPUs, GPUs, and FPGAs, used to perform various computing tasks. Storage nodes: These are responsible for data storage and management and can be SSDs, HDDs, or other types of storage devices. High-speed networks: These include high-speed network switches and other network components, used to connect compute and storage nodes.

[0030] Furthermore, intelligent distributed storage system:

[0031] Preferably, the intelligent data distribution module analyzes data access patterns and dynamically adjusts data locations to reduce read and write latency. Specific implementations are as follows:

[0032] Data access pattern analysis: Use the random forest algorithm to predict future data access patterns. Specifically, a classifier is trained to predict data access patterns. Let the input feature vector be X (e.g., recent access frequency, access time, etc.), and the output be the data access category (e.g., frequent access, occasional access, rare access).

[0033] Category = f(X)

[0034] Where f represents the random forest classifier function obtained through training.

[0035] Dynamic data migration: Based on the prediction results of the random forest algorithm, hot data is migrated to SSD or memory, and cold data is migrated to HDD or object storage.

[0036] Data distribution optimization: intelligently allocate data blocks to reduce the number of data accesses across nodes.

[0037] Preferably, the data redundancy and replication module ensures data security and high availability through data replication strategies and data recovery mechanisms.

[0038] Data replication strategy: Determine the number of copies of data based on its importance and access frequency.

[0039] Data recovery mechanism: When a node fails, the lost data can be quickly recovered.

[0040] Preferably, the data compression and encryption module reduces storage space requirements through efficient compression algorithms and uses data encryption technology to protect data security.

[0041] Efficient compression algorithm: Use the LZ77 compression algorithm to reduce storage space requirements and compress by finding repeated patterns in historical data. Suppose the input sequence S is compressed and the output is C;

[0042] C=LZ77Compress(S)

[0043] Data encryption: Use the AES encryption algorithm to protect the security of sensitive data. The AES encryption algorithm encrypts the plaintext P with the key K to obtain the ciphertext C:

[0044] C=AESEncrypt(P,K)

[0045] Furthermore, the adaptive load balancing framework:

[0046] Preferably, the load monitoring module monitors the usage of resources such as CPU and memory in real time, and tracks the status and progress of running tasks.

[0047] Preferably, the resource scheduling module dynamically adjusts computing resources according to real-time load conditions, and has a set of computing nodes N, each with its own computing capacity C i , where i∈N. Dynamically adjust the allocation of computing resources according to the real-time load situation L (for example, CPU usage, memory usage, etc.):

[0048]

[0049] Preferably, the task scheduling module: intelligently assigns tasks to the most appropriate computing nodes, with a task set T, each task has its priority P i , where i∈T. Intelligently allocate tasks based on the priority of the tasks and the real-time load of the computing nodes:

[0050]

[0051] The present invention has the following beneficial effects:

[0052] 1. The intelligent distributed storage system of this invention significantly reduces data access latency through intelligent data distribution strategies and improves system reliability and data security through data redundancy and replication. Furthermore, efficient data compression algorithms reduce the required storage space, thereby lowering storage costs.

[0053] 2. In this invention, the adaptive load balancing framework improves resource utilization and reduces task response time by dynamically adjusting computing resource allocation. Intelligent task scheduling ensures that tasks are processed efficiently, while improving service quality and system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a schematic diagram of the hardware architecture of a computer device based on big data processing proposed by the present invention;

[0055] Figure 2 This is a block diagram of the intelligent distributed storage system in the present invention;

[0056] Figure 3 This is a workflow diagram of the adaptive load balancing framework in the present invention;

[0057] Figure 4 This is a data processing flow chart of the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] Example 1

[0060] like Figure 1-Figure 4 As shown, the present invention proposes a computer device based on big data processing, including:

[0061] at least one computing node, the computing node including at least one high-performance central processing unit (CPU);

[0062] at least one storage node, the storage node including at least one high-speed solid-state drive (SSD);

[0063] An intelligent distributed storage system that dynamically adjusts data relocation based on data access patterns to reduce read and write latency;

[0064] An adaptive load balancing framework that dynamically adjusts the allocation of computing resources based on real-time system load;

[0065] A high-speed network that connects compute nodes and storage nodes.

[0066] In one embodiment, a computer device comprises:

[0067] Computing nodes: Contain computing resources such as high-performance CPUs, GPUs, and FPGAs, used to perform various computing tasks.

[0068] High-performance CPU: Intel Xeon E5-2600 v4 series, main frequency 2.2GHz, 16 cores.

[0069] GPU accelerator: NVIDIA Tesla P100 with 3584 CUDA cores.

[0070] FPGA accelerator: Xilinx Alveo U250, supporting high-speed data processing tasks.

[0071] Storage node: responsible for data storage and management, which can be SSD, HDD or other types of storage devices.

[0072] SSD solid state drive: Samsung PM1633a, capacity 3.2TB.

[0073] HDD mechanical hard drive: Seagate Exos X16, capacity 16TB.

[0074] High-speed network: High-speed network switches and other network components are used to connect various computing nodes and storage nodes.

[0075] High-speed Ethernet switch: Cisco Nexus 9396PX, supporting 100Gbps ports.

[0076] In one embodiment, the intelligent distributed storage system mainly includes:

[0077] Intelligent data distribution module: By analyzing data access patterns, it dynamically adjusts the location of data to reduce read and write latency. The specific implementation is as follows:

[0078] Data access pattern analysis: We use the RandomForestClassifier class in the Python Scikit-learn library to implement the random forest algorithm. The training dataset includes historical data access records, labeled as hotspot data.

[0079] Dynamic data migration: Based on the prediction results of the random forest algorithm, hot data is migrated to SSD or memory, and cold data is migrated to HDD or object storage. The random forest algorithm is implemented using the RandomForestClassifier class in the Python Scikit-learn library.

[0080] Data distribution optimization: Intelligently distribute data blocks to reduce the number of data accesses across nodes. Use hash functions to determine the optimal location of data blocks.

[0081] Data redundancy and replication module: Ensures data security and high availability through data replication strategies and data recovery mechanisms.

[0082] Data replication strategy: Determine the number of data replicas based on the importance and access frequency of the data. For example, for frequently accessed hot data, a higher number of replicas can be set to ensure that the data can still be accessed even if one or more nodes fail.

[0083] Data recovery mechanism: When a node fails, lost data can be quickly restored. Using heartbeat detection and a fault recovery mechanism, data is immediately restored from other replicas upon detecting a node failure. The heartbeat detection mechanism monitors node status by regularly sending heartbeat signals.

[0084] Data compression and encryption module: reduces storage space requirements through efficient compression algorithms and uses data encryption technology to protect data security.

[0085] Efficient compression algorithm: Uses the LZ77 compression algorithm to reduce storage space requirements. Compression is performed by finding repeated strings and replacing them with shorter pointers.

[0086] Data encryption: Use the cryptography library in Python to implement the AES encryption algorithm to protect the security of sensitive data and provide high-intensity security.

[0087] In one embodiment, the adaptive load balancing framework mainly includes:

[0088] Load monitoring module: monitors the usage of CPU, memory and other resources in real time, and tracks the status and progress of running tasks.

[0089] Resource usage monitoring: Use the Prometheus monitoring tool to monitor the usage of resources such as CPU, memory, and disk I / O in real time. Prometheus monitors resource usage by capturing indicator data on each node.

[0090] Task status tracking: Use the Prometheus monitoring tool to track the status and progress of running tasks. Prometheus monitors the status of tasks by scraping metric data from each node.

[0091] Resource scheduling module: dynamically adjusts computing resources based on real-time load conditions:

[0092]

[0093] Dynamic resource allocation: Use Kubernetes or other container orchestration tools to dynamically adjust computing resources based on real-time load conditions. Kubernetes can automatically manage the deployment, scaling, and operation of containerized applications.

[0094] Elastic Scaling: Automatically add compute nodes when the load increases and release resources when the load decreases. This is achieved using Kubernetes' autoscaling feature. Kubernetes can automatically increase or decrease the number of compute nodes based on pre-set rules to respond to changes in load.

[0095] Task scheduling module: intelligently assign tasks to the most appropriate computing nodes:

[0096]

[0097] Priority scheduling: Scheduling tasks based on their importance and urgency.

[0098] Load-aware scheduling: Use Kubernetes or other scheduling tools to assign tasks to nodes with lower loads. The Kubernetes scheduler determines which node to schedule the task to based on the node's resource usage.

[0099] In one embodiment, the device workflow is:

[0100] Data collection and preprocessing: Collect raw data from various sources and perform preliminary cleaning and preprocessing.

[0101] Data storage: The processed data is stored on storage nodes according to the intelligent data distribution strategy.

[0102] Task submission: Users submit big data processing tasks through the user interface.

[0103] Resource scheduling: The adaptive load balancing framework allocates appropriate computing resources to tasks based on the current system load.

[0104] Data processing: Compute nodes perform assigned tasks and store the results back to storage nodes.

[0105] Result presentation: After processing is completed, the results can be viewed or exported through the user interface.

[0106] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A computer device based on big data processing, characterized in that: include: A computing node, wherein the computing node comprises at least one central processing unit (CPU); a storage node, wherein the storage node comprises at least one solid-state drive (SSD); Intelligent distributed storage system to dynamically adjust data location based on data access patterns; Adaptive load balancing framework for dynamically adjusting the allocation of computing resources based on real-time system load; A high-speed network, used to connect the computing node and the storage node; The computing node obtains data from the storage node through a high-speed network, executes the computing task, and then writes the result back to the storage node. The intelligent distributed storage system monitors the data access pattern on the storage node and adjusts the position of the data between different storage media through the high-speed network according to the prediction results. The adaptive load balancing framework monitors the load of the computing node in real time through the load monitoring module, and dynamically adjusts the allocation of computing resources through the resource scheduling module and the task scheduling module.

2. A computer device based on big data processing according to claim 1, characterized in that: The intelligent distributed storage system includes a data redundancy and replication module for determining the number of copies of data according to the importance and access frequency of the data; and a data compression and encryption module for reducing the demand for storage space and protecting the security of data.

3. The computer device based on big data processing according to claim 1, characterized in that: The adaptive load balancing framework includes: a load monitoring module for monitoring the resource usage of the computing nodes in real time; a resource scheduling module for dynamically adjusting the allocation of computing resources according to the real-time load situation; and a task scheduling module for prioritizing tasks according to their attributes and allocating tasks to the most suitable computing nodes.

4. The computer device based on big data processing according to claim 1, characterized in that: The intelligent distributed storage system includes: a data access pattern analysis module, which is used to predict future data access patterns through machine learning algorithms; and a dynamic data migration module, which is used to migrate hot data to solid-state drives (SSDs) and cold data to hard disk drives (HDDs) based on the results of the data access pattern analysis module.

5. The computer device based on big data processing according to claim 1, characterized in that: The adaptive load balancing framework includes: an elastic scaling module for automatically adding computing nodes when the load increases and releasing computing resources when the load decreases.

6. The computer device based on big data processing according to claim 1, characterized in that: The computing nodes include graphics processing units (GPUs) and field programmable gate arrays (FPGAs) to accelerate specific types of data processing tasks.

7. The computer device based on big data processing according to claim 1, characterized in that: The storage node includes an object storage service for storing large amounts of data at low cost.

8. A method for processing big data based on a computer device for big data processing according to any one of claims 1 to 7, characterized in that: The following steps are involved: Step A: Using the intelligent distributed storage system to dynamically adjust the location of data according to the data access pattern; Step B: using the adaptive load balancing framework to dynamically adjust the allocation of computing resources according to the real-time system load; Step C: executing the big data processing task through the computing node; Step D: Storing the processing result through the storage node.

9. The method for processing big data based on a computer device for big data processing according to claim 8, characterized in that: The step A further comprises: Use the data access pattern analysis module to predict future data access patterns; Use the dynamic data migration module to migrate hot data to SSD or memory and cold data to HDD or object storage based on prediction results.

10. The method for processing big data based on a computer device for big data processing according to claim 1, characterized in that: The step B further comprises: Use the load monitoring module to monitor the resource usage of computing nodes in real time; Use the resource scheduling module to dynamically adjust the allocation of computing resources according to real-time load conditions; Use the task scheduling module to prioritize tasks according to their attributes and assign them to the most suitable computing nodes.

Images