Computer cluster data processing method and processing device

Abstract

The application relates to the technical field of data processing, and discloses a computer cluster data processing method and device, the method comprises the following steps: acquiring cluster data, wherein the cluster data is any one of continuous storage data and dispersed storage data; performing chain layer storage and analysis on a computer cluster according to the cluster data to obtain a cluster data block; wherein the computer cluster refers to a computer executable program running on a cluster storage structure; performing vectorization coding according to the cluster data block to obtain a data block vector; performing mapping according to the data block vector to obtain mapping data which can be operated on a computer; and inputting the mapping data into a data calculation formula to perform calculation processing, so as to obtain a processed data linked list. The method can solve the problem of low data processing efficiency in the prior art due to the limited capacity of structured modeling for mass data processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a computer cluster data processing method and processing device. Background Technology

[0002] Currently, in today's Internet+Big Data era, the demand for data processing is increasing, the computing requirements are becoming more complex, and the requirements for data processing capabilities and efficiency are becoming higher.

[0003] In one existing technology, data is processed by using middleware or dedicated equipment (controllers), while structured modeling supports the data processing.

[0004] Existing technologies suffer from low data processing efficiency due to the limited capacity of structured modeling to process massive amounts of data. Summary of the Invention

[0005] This invention provides a computer cluster data processing method and apparatus to solve the problem of low data processing efficiency due to the limited ability of structured modeling to process massive amounts of data.

[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a computer cluster data processing method, comprising:

[0007] Acquire cluster data, wherein the cluster data refers to either continuously stored data or distributed stored data;

[0008] The cluster data is stored and analyzed in a chained hierarchical manner on a computer cluster to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure on which computer executable programs are running.

[0009] The cluster data blocks are vectorized and encoded to obtain data block vectors;

[0010] Mapping is performed based on data block vectors to obtain mapped data that can be processed on a computer;

[0011] The mapped data is input into the data calculation formula for processing, resulting in a processed data linked list.

[0012] In one optional implementation, the step of vectorizing the cluster data blocks to obtain data block vectors includes:

[0013] Extract data from the cluster data blocks to obtain a linked list of data blocks;

[0014] The linked list of data blocks is decomposed to obtain linked list data blocks of equal or unequal length.

[0015] The linked list data blocks are encoded to obtain a data block vector.

[0016] In one optional implementation, the step of performing chained hierarchical storage and analysis on the computer cluster based on the cluster data to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure running computer-executable programs, including:

[0017] Obtain the number of computers, storage size, and data encoding bit width in the computer cluster;

[0018] Based on a comprehensive analysis of the number of computers, the storage size, and the number of data encoding bits, the data block storage space data is obtained.

[0019] The hash function is obtained by performing calculations and analysis based on the storage space data of the data block;

[0020] The maximum chain length of the data block is calculated using the chain length dynamic adjustment algorithm based on the prediction model according to the hash function.

[0021] The cluster data blocks are allocated based on the maximum chain length of the data blocks using a greedy algorithm.

[0022] In one optional implementation, the step of inputting the mapped data into a data calculation formula for calculation and processing to obtain a processed data linked list includes:

[0023] Based on the mapping data, the calculation operations for adding data, deleting data, and updating data are determined.

[0024] When the required operation is the calculation operation of the data to be added, the mapping data is input into the calculation formula of the added data to calculate and obtain the added data. The linked list is updated according to the added data to obtain the added data linked list.

[0025] When the required operation is a calculation operation of the data to be deleted, the mapping data is input into the data deletion calculation formula for calculation to obtain the deleted data, and the linked list is updated according to the deleted data to obtain the deleted data linked list;

[0026] When the required operation is a calculation operation of the data to be updated, the mapping data is input into the update data calculation formula for calculation to obtain the updated data, and the linked list is updated according to the updated data to obtain the updated data linked list.

[0027] In one optional implementation, when the required operation is a calculation operation to add data, the mapped data is input into the data addition calculation formula for calculation to obtain the added data, and the linked list is updated according to the added data to obtain the added data linked list, including:

[0028] The added data is calculated in the following way:

[0029]

[0030] in, This refers to the i-th piece of data being added, where n is the number of data blocks in the cluster. It is the j-th mapping data that needs to be added.

[0031] In one optional implementation, when the required operation is a calculation operation on the data to be deleted, the mapping data is input into the data deletion calculation formula for calculation to obtain the deleted data, and the linked list is updated according to the deleted data to obtain the deleted data linked list, including:

[0032] Calculate deleted data using the following method:

[0033]

[0034] in, This refers to the i-th deleted data, where n is the number of data blocks in the cluster. It is the j-th mapping data to be deleted.

[0035] In one optional implementation, when the required operation is a calculation operation of the data to be updated, the mapping data is input into the update data calculation formula for calculation to obtain the updated data, and the linked list is updated according to the updated data to obtain the updated data linked list, including:

[0036] The updated data is calculated in the following way:

[0037]

[0038] in, This is the i-th updated data, where n is the number of data blocks in the cluster. This is the j-th mapping data to be updated. Secondly, the present invention provides a computer cluster data processing apparatus, comprising:

[0039] The data acquisition module is used to acquire cluster data, wherein the cluster data refers to either continuously stored data or distributed stored data.

[0040] The data storage module is used to perform chained hierarchical storage and analysis on the computer cluster based on the cluster data to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure on which a computer executable program is running;

[0041] The data encoding module is used to perform vectorized encoding on the cluster data blocks to obtain data block vectors;

[0042] The data mapping module is used to map data based on data block vectors to obtain mapped data that can be processed on a computer.

[0043] The data calculation module is used to input the mapped data into the data calculation formula for calculation and processing, and to obtain the processed data linked list.

[0044] Thirdly, the present invention also provides an electronic device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the computer cluster data processing method described in any one of the above.

[0045] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the computer cluster data processing method described in any one of the above.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] This invention discloses a computer cluster data processing method, comprising: acquiring cluster data, wherein the cluster data refers to either continuously stored data or distributed stored data; performing chained hierarchical storage and analysis on the computer cluster based on the cluster data to obtain cluster data blocks; wherein the computer cluster refers to a cluster storage structure on which a computer-executable program runs; performing vectorized encoding on the cluster data blocks to obtain data block vectors; mapping based on the data block vectors to obtain mapped data that can be computed on the computer; and inputting the mapped data into a data calculation formula for calculation and processing to obtain a processed data linked list. This invention supports large-scale data cluster storage (including file, video data, and unstructured relational databases, distributed database storage) based on data block chained storage technology, and uses a computer cluster architecture to complete large-scale data processing calculations, fully utilizing structured modeling or the multiple computing nodes of a distributed computer cluster to support large-scale data processing. Compared to existing technologies that use middleware or dedicated devices (controllers) to process data while simultaneously using structured modeling to support data processing, this invention solves the problem of low data processing efficiency due to the limited capacity of structured modeling for processing massive amounts of data. Attached Figure Description

[0048] Figure 1 This is a schematic flowchart of the computer cluster data processing method provided in the first embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the computer node data processing flow provided by the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of a computer cluster data processing device provided in the second embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Reference Figure 1 The first embodiment of the present invention provides a computer cluster data processing method, including the following steps:

[0053] S11, Obtain cluster data, wherein the cluster data refers to either continuously stored data or distributed stored data;

[0054] S12, based on the cluster data, perform chained hierarchical storage and analysis on the computer cluster to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure on which a computer executable program is running;

[0055] S13, Perform vectorization encoding on the cluster data blocks to obtain data block vectors;

[0056] S14, Map the data block vector to obtain the mapped data that can be processed on a computer;

[0057] S15, the mapping data is input into the data calculation formula for calculation and processing to obtain the processed data linked list.

[0058] In step S11, cluster data is obtained, wherein the cluster data refers to either continuously stored data or distributed stored data.

[0059] Continuous data storage refers to data stored sequentially in a specific order on a physical storage medium. This data storage method is suitable for scenarios requiring fast sequential access, such as video files or certain tables in a large database. Because the data is continuous on the physical storage medium, sequential read and write operations can be performed efficiently. Continuously stored data is relatively simple to manage because its physical location facilitates data backup, recovery, and maintenance. Distributed data storage refers to data distributed across different storage nodes on a physical storage medium. This storage method is suitable for distributed systems, such as distributed file systems and distributed databases. Data is distributed across multiple nodes, so even if one node fails, the data can still be accessed through other nodes. By adding storage nodes, storage capacity can be easily expanded, improving the overall system performance. Distributed data storage can achieve load balancing, avoiding overload of a single node and improving overall system performance. For example, the Hadoop Distributed File System (HDFS) distributes data blocks across multiple nodes, supporting efficient storage and access to large-scale data; Cassandra distributes data across multiple nodes, supporting high concurrency access and high data availability.

[0060] Data acquisition can involve data generated internally within the cluster, such as server logs and user behavior data, or data obtained from external systems, such as data crawled by web crawlers or provided by external APIs. Data acquisition methods can include periodically retrieving data in batches from data sources, for example, using tools like Flume and Sqoop for periodic batch data collection. Flume can batch collect log data to HDFS, and Sqoop can batch import data from relational databases to HDFS. This method is suitable for scenarios where data updates are infrequent. Alternatively, data can be acquired in real-time using data streaming technologies, such as using tools like Kafka and Logstash for real-time data collection. Kafka can efficiently handle massive data streams, and Logstash can collect and process log data in real time. This method is suitable for data scenarios requiring real-time processing, such as real-time monitoring systems and real-time transaction systems.

[0061] Data transmission can be performed on the cluster data to the storage nodes via network. Protocols such as TCP / IP and HTTP can be used for this purpose. The TCP / IP protocol uses a multi-layered encapsulation and decapsulation process to ensure accurate data transmission from the sender to the receiver. Each layer's header contains necessary control information to ensure data integrity, order, and security. Alternatively, cluster data can be transmitted to the storage nodes locally. If the data source and storage nodes are on the same local area network, a local file system or high-speed network can be used for data transmission.

[0062] In step S12, the cluster data is stored and analyzed in a chained hierarchical manner on the computer cluster to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure on which a computer executable program is running.

[0063] In one implementation, the number of computers, storage size, and data encoding bit width in the computer cluster are obtained; a comprehensive analysis is performed based on the number of computers, the storage size, and the data encoding bit width to obtain data block storage space data; a hash function is obtained by calculation and analysis based on the data block storage space data; the maximum data block chain length is obtained by calculating the chain length dynamic adjustment algorithm based on the hash function and a prediction model; and cluster data blocks are allocated based on the maximum data block chain length combined with a greedy algorithm.

[0064] The number of computers refers to the total number of computers (nodes) in the cluster. It determines the cluster size to facilitate the distribution of data processing across computing nodes and parallel processing capabilities. The total number of computers (nodes) in the cluster can be obtained through cluster management tools (such as Hadoop's ResourceManager) or by directly querying the cluster configuration file. Storage size refers to the storage capacity of each node. It determines the amount of data each node can store, thus determining the size and number of data blocks. Storage size data can be obtained through system commands (such as `df -h`) or by querying the cluster management tool. Data encoding bit width refers to the number of bits used in data encoding. This bit width is related to the uniqueness of the data and the generation of the hash function. The data encoding bit width is used for hash function generation and to verify the uniqueness of data blocks, ensuring a uniform distribution of data in the cluster. The data encoding bit width can be obtained based on the characteristics of the data and storage requirements. The data encoding bit width is set during the system design phase. The number of data blocks is determined during the comprehensive analysis of the data block storage space data. The number of data blocks refers to the total number of data blocks in the cluster. It can be calculated using a preset formula, which divides the total storage capacity of the cluster nodes by the data block size. For example, assuming a total storage capacity of 5TB and a data block size of 128MB, the calculated number of data blocks would be 40,000. The calculation is performed based on the data block storage space data, and a hash function is selected accordingly. A hash function is a function used to evenly distribute data blocks across different nodes in the cluster, ensuring uniform data distribution, reducing hotspot issues, and improving system availability and performance. A suitable hash function is selected based on the characteristics and storage requirements of the data blocks. Commonly used hash functions include MD5 (generating a 128-bit hash value, suitable for smaller data blocks), SHA-1 (generating a 160-bit hash value, suitable for larger data blocks), and MurmurHash (generating a 128-bit hash value, with higher performance, suitable for large-scale data processing). This embodiment of the invention does not limit the specific hash function used.

[0065] A predictive model is a model used to predict data access frequency and storage node load. It can dynamically adjust the length of the data block chain to optimize storage and access performance. Predictive models include time series analysis models and machine learning models. Time series analysis models are suitable for predicting data access frequency, such as the ARIMA model. Machine learning models are suitable for predicting storage node load, such as linear regression and decision trees. The chain length dynamic adjustment algorithm dynamically adjusts the length of the data block chain based on the results of the predictive model. Assuming the chain length output by the predictive model is 10 and the adjustment coefficient is 1.2, the algorithm determines that the maximum chain length is 12. A greedy algorithm is one that makes the best or optimal choice at each step in the current state. The result is globally best or optimal. Greedy algorithms can ensure efficient distribution of data blocks in the cluster and reduce data access latency. Greedy algorithms include minimum load first algorithm and shortest path first algorithm. The minimum load first algorithm selects the node with the lowest load to store the data block, while the shortest path first algorithm selects the node with the shortest path to store the data block. The allocation is performed based on the maximum chain length of the data blocks using a greedy algorithm. First, the current load of each node is recorded. Then, for each data block, the node with the lowest load is selected for storage. Finally, the node load information is updated. For example, if node 1 has a load of 10%, node 2 has a load of 15%, node 3 has a load of 12%, node 4 has a load of 18%, and node 5 has a load of 14%, then data block 1 is allocated to node 1, data block 2 to node 3, data block 3 to node 1, and so on. Finally, each node is updated: node 1 load 12%, node 2 load 15%, node 3 load 14%, node 4 load 18%, and node 5 load 14%.

[0066] In step S13, the cluster data blocks are vectorized and encoded to obtain data block vectors.

[0067] In one implementation, the cluster data blocks are extracted to obtain a linked list of data blocks; the linked list of data blocks is decomposed to obtain linked list data blocks of equal or unequal length; and the linked list data blocks are encoded to obtain a data block vector.

[0068] A data block linked list is a data structure where each data block contains two parts: the data content and a pointer to the next data block. This structure allows data blocks to be linked together linearly to form a linked list. This structure facilitates sequential access and management of data, supports dynamic insertion and deletion operations, and is particularly suitable for distributed storage systems, allowing for flexible handling of data block distribution and movement. First, each node in the cluster is traversed to extract the data blocks stored on that node and connect them in order to form a linked list. Starting from the first node of the cluster, each node is traversed sequentially. At each node, all stored data blocks are extracted and connected in their storage order within the node to form a complete linked list. For example, assuming node 1 stores data blocks [B1, B2, B3], node 2 stores data blocks [B4, B5], and node 3 stores data block [B6], then the data block linked list would be [B1 -> B2 -> B3 -> B4 -> B5 -> B6].

[0069] A linked list data block is a single data block decomposed from a linked list of data blocks. These blocks can be of equal or unequal length. Decomposing the data block into individual units facilitates individual processing and encoding of each block. Decomposing a linked list of data blocks into data blocks of equal length is suitable for cases where the data block size is fixed. First, determine the fixed length of the data blocks. Then, starting from the head of the list, extract data blocks of fixed length sequentially. If the data block at the end of the list is shorter than the fixed length, it is processed as needed (e.g., discarded, padded, etc.). For example, assuming the linked list of data blocks is [B1 -> B2 -> B3 -> B4 -> B5 -> B6], and each data block is 128MB, then the decomposed data blocks of equal length are [B1, B2, B3, B4, B5, B6]. Decomposing a linked list of data blocks into data blocks of unequal length is suitable for cases where the data block size is not fixed. First, start from the head of the list, extract each data block sequentially, record the actual length of each data block, and finally treat each data block and its length as a separate unit. For example, assuming the linked list of data blocks is [B1 -> B2 -> B3 -> B4 -> B5 -> B6], and the sizes of the data blocks are 128MB, 256MB, 128MB, 128MB, 256MB, and 128MB respectively, then the decomposed linked list of data blocks of unequal lengths is [B1, B2, B3, B4, B5, B6].

[0070] A data block vector is a vector representation of linked list data blocks. Each element of the vector represents a feature of the data block. Obtaining data block vectors facilitates subsequent mathematical modeling and computation, and supports parallel data processing. The process involves feature extraction and encoding for each linked list data block. First, the byte size of the data block, the specific data within it (which can be numerical, text, binary, etc.), and the block's position in the linked list are determined to identify its order. Encoding then proceeds. First, the content of the data block is extracted and converted into numerical form. For text data, ASCII values ​​of the characters are used; for numerical data, the numerical value is used directly; and for binary data, binary representation or conversion to numerical values ​​is used. For example, for text data, ASCII values ​​of the characters can be used. Next, the position of each data block in the linked list is determined and converted into numerical form. For example, position 1 can be encoded as 1, position 2 as 2, and so on. Finally, the size of each data block is determined and converted into numerical form. For example, 128MB can be encoded as 128, and 256MB as 256. The size, content, and position of a data block are encoded and combined into a single vector. For example, if data block B1 is 128MB in size, contains [1, 2, 3, 4], and is located at position 1, then the data block vector F1 is [128, 1, 2, 3, 4, 1], thus obtaining the final data block vector.

[0071] In step S14, the data block vector is mapped to obtain mapped data that can be processed on a computer.

[0072] Mapping refers to transforming a data block vector from one representation to another, making it more suitable for mathematical operations and data processing by computers. The purpose of mapping is to convert complex data structures or high-dimensional data into simple numerical forms, facilitating efficient computer processing and computation. Through mapping, complex data structures (such as text, images, and audio) can be converted into numerical vectors for easier computer processing. It also converts data into a format suitable for computer operations, reducing computational complexity and improving processing speed. Furthermore, mapping transforms data into a more understandable and interpretable form, facilitating data analysis and modeling. Linear mapping can be performed using matrix multiplication. Matrix multiplication involves multiplying a data block vector by a predefined matrix to obtain a new vector. Matrix multiplication is a linear transformation that can be used for dimensionality reduction, feature extraction, and data transformation. A predefined matrix A is chosen, and its dimensions should be compatible with the dimensions of the data block vector. For example, if the dimension of the data block vector is n, then the dimension of matrix A should be m × n, where m is the dimension of the mapped vector. Multiplying the data block vector F by matrix A yields a new vector F'. Linear mapping can also be performed using vector addition. Vector addition involves adding a new vector to a predefined vector, further adjusting the vector's value. Vector addition can be used for data offsetting and adjustment. A predefined vector *b* is chosen, and its dimension should be the same as the dimension of the mapped vector. For example, if the dimension of the mapped vector is *m*, then the dimension of vector *b* should also be *m*. The new vector *F'* is added to vector *b* to obtain the final mapped vector *F''*. Nonlinear mapping can also be used, including activation functions and polynomial functions. Activation functions transform each element of the data block vector through a nonlinear function. Commonly used activation functions include ReLU, Sigmoid, and Tanh. Activation functions can introduce nonlinear factors, enabling the model to learn more complex features. First, a suitable activation function, such as ReLU, Sigmoid, or Tanh, is selected and applied to each element of the data block vector to obtain a new vector. Polynomial functions transform each element of the data block vector through a polynomial. Polynomial functions can introduce higher-order features, enabling the model to learn more complex patterns. Choose a suitable polynomial function, such as a quadratic or cubic polynomial, and apply it to each element of the data block vector to obtain a new vector. The mapped data block vector is then converted into a numerical vector, where each element is numerical. Numerical vectors facilitate efficient mathematical operations by computers, such as matrix multiplication, vector addition, and dot product. The mapped data block vector can also be used for data processing tasks, such as data cleaning, feature extraction, and data fusion.

[0073] In step S15, the mapping data is input into the data calculation formula for calculation and processing to obtain the processed data linked list.

[0074] In one implementation, based on the mapping data, calculation operations for adding data, deleting data, and updating data are determined. When the required operation is the calculation operation for adding data, the mapping data is input into the data addition calculation formula for calculation, resulting in the added data. The linked list is then updated based on the added data to obtain the added data linked list. When the required operation is the calculation operation for deleting data, the mapping data is input into the data deletion calculation formula for calculation, resulting in the deleted data. The linked list is then updated based on the deleted data to obtain the deleted data linked list. When the required operation is the calculation operation for updating data, the mapping data is input into the data update calculation formula for calculation, resulting in the updated data. The linked list is then updated based on the updated data to obtain the updated data linked list.

[0075] The added data is calculated in the following way:

[0076]

[0077] in, This refers to the i-th piece of data being added, where n is the number of data blocks in the cluster. It is the j-th mapping data that needs to be added.

[0078] Calculate deleted data using the following method:

[0079]

[0080] in, This refers to the i-th deleted data, where n is the number of data blocks in the cluster. It is the j-th mapping data to be deleted.

[0081] The updated data is calculated in the following way:

[0082]

[0083] in, This is the i-th updated data, where n is the number of data blocks in the cluster. This is the j-th mapping data to be updated. Based on the characteristics or content of the mapping data, the type of operation to be performed (add, delete, or update) is determined. After determining the specific data processing operation, an appropriate calculation formula is selected for processing. During the determination process, the characteristics of the mapping data, such as the size, content, and position of the data block, need to be analyzed to determine the operation to be performed. Based on predefined rules or conditions, the operation required for the data block is determined. For example, if a certain characteristic value of the data block exceeds a threshold, an add operation is performed; if a certain characteristic value of the data block is 0, a delete operation is performed; if a certain characteristic value of the data block changes, an update operation is performed. When an add data operation is required, an empty add data linked list is initialized to temporarily store the data block to be added. The add data is calculated using the calculation formula, the new add data block is added to the existing data linked list, the data linked list is expanded, new data blocks are added, the linked list is updated, and so on. When a data deletion operation is required, an empty deletion list is initialized to temporarily store the data blocks to be deleted. The sum of each mapped data block is multiplied by a deletion flag, and the data to be deleted is calculated using a formula. The specific data block to be deleted is then removed from the existing data list, and the list is updated. Conversely, when a data deletion operation is required, an empty update list is initialized to temporarily store the data to be updated. The sum of each mapped data block is multiplied by an update flag, and the data block to be updated is calculated using a formula. The specific data block in the existing data list is then updated, modifying the data blocks in the data list to reflect the latest data state.

[0084] To facilitate understanding of the present invention, some preferred embodiments of the present invention will be described in further detail below.

[0085] The following describes the working process of this invention using a common scenario as an example. Please also refer to... Figure 2 , it is Figure 1 A schematic diagram of the working scenario of the method.

[0086] In modern distributed systems, data storage and processing are critical issues. Assume a distributed file system composed of multiple nodes, each responsible for storing and processing a portion of the data. To improve the efficiency and reliability of data processing, a computer cluster data processing method based on data block linked storage and vectorized encoding is adopted. First, cluster data needs to be obtained from the distributed file system. Assume there are 5 nodes, and the data blocks stored on each node are as follows: Node 1: [B1, B2, B3], Node 2: [B4, B5], Node 3: [B6], Node 4: [B7, B8], Node 5: [B9]. These data blocks can be stored contiguously or distributed. Assume the data blocks are distributed, and the size of each data block is not fixed. Next, these data blocks are stored in a linked hierarchical manner and analyzed to obtain the cluster data blocks. The specific steps are as follows: 1. Extract the data block linked list: Traverse each node, extract the data blocks stored on that node, and connect them in order to form a linked list. The result is [B1 -> B2 -> B3 -> B4 -> B5 -> B6 -> B7 -> B8 -> B9]. 2. Decompose the linked list data blocks: Decompose the linked list of data blocks into data blocks of equal or unequal length. Assume the sizes of the data blocks are 128MB, 256MB, 128MB, 128MB, 256MB, 128MB, 128MB, 256MB, and 128MB respectively. The result is [B1, B2, B3, B4, B5, B6, B7, B8, B9]. Perform feature extraction and encoding on each linked list data block to generate a data block vector. The specific steps are as follows: Feature extraction: Extract the features of the data blocks, such as their size, content, and position. For example, B1: size 128MB, content [1, 2, 3, 4], position 1. B2: size 256MB, content [5, 6, 7, 8], position 2. B3: Size 128MB, Content [9, 10, 11, 12], Position 3. B4: Size 128MB, Content [13, 14, 15, 16], Position 4. B5: Size 256MB, Content [17, 18, 19, 20], Position 5. B6: Size 128MB, Content [21, 22, 23, 24], Position 6. B7: Size 128MB, Content [25, 26, 27, 28], Position 7. B8: Size 256MB, Content [29, 30, 31, 32], Position 8. B9: Size 128MB, Content [33, 34, 35, 36], Position 9. Encoding: Convert the features into numerical form to generate a data block vector.The results are: F1: [128, 1, 2, 3, 4, 1], F2: [256, 5, 6, 7, 8, 2], F3: [128, 9, 10, 11, 12, 3], F4: [128, 13, 14, 15, 16, 4], F5: [256, 17, 18, 19, 20, 5], F6: [128, 21, 22, 23, 24, 6], F7: [128, 25, 26, 27, 28, 7], F8: [256, 29, 30, 31, 32, 8], F9: [128, 33, 34, 35, 36, 9]. Mapping data from data block vectors yields mapped data that can be processed on a computer. The specific steps are as follows: Linear mapping: Perform linear mapping using a predefined matrix A and vector b. The mapped vectors are F1': [128, 1, 2, 3, 4, 1], F2': [256, 5, 6, 7, 8, 2], F3': [128, 9, 10, 11, 12, 3], F4': [128, 13, 14, 15, 16, 4], F5': [256, 17, 18, 19, 20, 5], F6': [128, 21, 22, 23, 24, 6], F7': [128, 25, 26, 27, 28, 7], F8': [256, 29, 30, 31, 32, 8], F9': [128, 33, 34, 1]. [35, 36, 9]. Determine the operation type: Based on the characteristics or content of the mapped data, determine the type of operation to be performed (add, delete, or update). Specific steps are as follows: Feature Analysis: Analyze the characteristics of the mapped data, such as the size, content, and location of the data blocks, to determine the operation to be performed. If the data block size exceeds 200MB, perform an add operation. If a specific value in the content of the data block is 0, perform a delete operation. If the location of the data block changes, perform an update operation. Determine the operation required for the data block based on predefined rules or conditions. Add rule: If the data block size exceeds 200MB, add a new data block. Delete rule: If a specific value in the content of the data block is 0, delete the data block. Update rule: If the location of the data block changes, update the content of the data block. Based on the determination result, perform the corresponding calculation operation (add, delete, or update data). Assume that one data block in the mapped data exceeds 200MB in size and a new data block needs to be added.Mapped data: [256, 5, 6, 7, 8, 2] (assuming this is data block B2), add a new data block B10 (content 3072) to the end of the linked list: [B1 -> B2 -> B3 -> B4 -> B5 -> B6 -> B7 -> B8 -> B9 -> B10]. Assume a data block in the mapped data has a specific value of 0, and this data block needs to be deleted. Mapped data: [128, 0, 2, 3, 4, 6] (assuming this is data block B6), deletion flag: [0, 0, 0, 0, 0, 1, 0, 0, 0] (indicating deletion of B6), delete data block B6 from the linked list: [B1->B2 -> B3 -> B4 -> B5 -> B7 -> B8 -> B9 -> B10]. Assume the position of a data block in the mapped data changes, and the content of this data block needs to be updated. Mapped data: [128, 1, 2, 3, 4, 3] (assuming this is data block B3), update flag: [0, 0, 1, 0, 0, 0, 0, 0, 0] (indicating updating B3), update the content of data block B3 to 2: [B1 ->B2 ->B3(2) ->B4 ->B5 ->B7 ->B8 ->B9 ->B10]. Update the data list according to the calculation results (add, delete, update) to ensure that the data list reflects the latest data state, add the new data block to the end of the list or the specified position. Delete the specified data block from the list. Replace the specified data block in the list with the new data block.

[0087] In summary, this invention discloses a computer cluster data processing method, comprising: acquiring cluster data, wherein the cluster data refers to either continuously stored data or distributed stored data; performing chained hierarchical storage and analysis on a computer cluster based on the cluster data to obtain cluster data blocks; wherein the computer cluster refers to a cluster storage structure on which a computer-executable program runs; performing vectorized encoding on the cluster data blocks to obtain data block vectors; mapping based on the data block vectors to obtain mapped data that can be computed on a computer; and inputting the mapped data into a data calculation formula for calculation and processing to obtain a processed data linked list. This invention supports large-scale data cluster storage (including file, video data, and unstructured relational databases, distributed database storage) based on data block chained storage technology, and uses a computer cluster architecture to complete large-scale data processing calculations, fully utilizing structured modeling or the multiple computing nodes of a distributed computer cluster to support large-scale data processing. Compared to existing technologies that use middleware or dedicated devices (controllers) to process data while simultaneously using structured modeling to support data processing, this invention solves the problem of low data processing efficiency due to the limited capacity of structured modeling for processing massive amounts of data.

[0088] Reference Figure 3 The second embodiment of the present invention provides a computer cluster data processing device, comprising:

[0089] The data acquisition module is used to acquire cluster data, wherein the cluster data refers to either continuously stored data or distributed stored data.

[0090] The data storage module is used to perform chained hierarchical storage and analysis on the computer cluster based on the cluster data to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure on which a computer executable program is running;

[0091] The data encoding module is used to perform vectorized encoding on the cluster data blocks to obtain data block vectors;

[0092] The data mapping module is used to map data based on data block vectors to obtain mapped data that can be processed on a computer.

[0093] The data calculation module is used to input the mapped data into the data calculation formula for calculation and processing, and to obtain the processed data linked list.

[0094] It should be noted that the computer cluster data processing device provided in this embodiment of the invention is used to execute all the process steps of the computer cluster data processing method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0095] This invention also provides an electronic device. The electronic device includes a processor, a memory, and a computer program, such as a computer algorithm program, stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps described in the various computer cluster data processing method embodiments above, for example... Figure 1 The step S11 shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, such as the data calculation module.

[0096] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0097] The electronic device may be a desktop computer, laptop, handheld computer, or smart tablet, etc. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0098] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting all parts of the electronic device via various interfaces and lines.

[0099] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0100] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0101] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0102] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A computer cluster data processing method, characterized in that, Executed by a computer, including: Acquire cluster data, wherein the cluster data refers to either continuously stored data or distributed stored data; The cluster data is stored and analyzed in a chained hierarchical manner on a computer cluster to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure on which computer executable programs are running. The cluster data blocks are vectorized and encoded to obtain data block vectors; Mapping is performed based on data block vectors to obtain mapped data that can be processed on a computer; The mapped data is input into a data calculation formula for processing, resulting in a processed data linked list, specifically including: Based on the mapping data, calculation operations for adding data, deleting data, and updating data are determined. When the required operation is the data deletion operation, the mapping data is input into the data deletion calculation formula for calculation, resulting in the deleted data. The linked list is then updated based on the deleted data to obtain the deleted data linked list. When the required operation is the data update operation, the mapping data is input into the data update calculation formula for calculation, resulting in the updated data. The linked list is then updated based on the updated data to obtain the updated data linked list. When the required operation is the data addition operation, the mapping data is input into the data addition calculation formula for calculation, resulting in the added data. The linked list is then updated based on the added data to obtain the added data linked list. The data addition calculation formula is as follows: in, This refers to the i-th piece of data being added, where n is the number of data blocks in the cluster. It is the j-th mapping data that needs to be added.

2. The computer cluster data processing method according to claim 1, characterized in that, The step of vectorizing the cluster data blocks to obtain data block vectors includes: Extract data from the cluster data blocks to obtain a linked list of data blocks; The linked list of data blocks is decomposed to obtain linked list data blocks of equal or unequal length. The linked list data blocks are encoded to obtain a data block vector.

3. The computer cluster data processing method according to claim 1, characterized in that, The process involves performing chained hierarchical storage and analysis on the computer cluster based on the cluster data to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure running computer-executable programs, including: Obtain the number of computers, storage size, and data encoding bit width in the computer cluster; Based on a comprehensive analysis of the number of computers, the storage size, and the number of data encoding bits, the data block storage space data is obtained. The hash function is obtained by performing calculations and analysis based on the storage space data of the data block; The maximum chain length of the data block is calculated using the chain length dynamic adjustment algorithm based on the prediction model according to the hash function. The cluster data blocks are allocated based on the maximum chain length of the data blocks using a greedy algorithm.

4. The computer cluster data processing method according to claim 1, characterized in that, When the required operation is a calculation operation on the data to be deleted, the mapping data is input into the data deletion calculation formula for calculation to obtain the deleted data. The linked list is then updated based on the deleted data to obtain the deleted data linked list, including: Calculate deleted data using the following method: in, This refers to the i-th deleted data, where n is the number of data blocks in the cluster. It is the j-th mapping data to be deleted.

5. The computer cluster data processing method according to claim 1, characterized in that, When the required operation is a calculation operation on the data to be updated, the mapping data is input into the update data calculation formula for calculation to obtain the updated data. The linked list is then updated based on the updated data to obtain the updated data linked list, including: The updated data is calculated in the following way: in, This is the i-th updated data, where n is the number of data blocks in the cluster. It is the j-th mapping data to be updated.

6. A computer cluster data processing device, characterized in that, include: The data acquisition module is used to acquire cluster data, wherein the cluster data refers to either continuously stored data or distributed stored data. The data storage module is used to perform chained hierarchical storage and analysis on the computer cluster based on the cluster data to obtain cluster data blocks; wherein, the computer cluster refers to a cluster storage structure on which a computer executable program is running; The data encoding module is used to perform vectorized encoding on the cluster data blocks to obtain data block vectors; The data mapping module is used to map data based on data block vectors to obtain mapped data that can be processed on a computer. The data calculation module is used to input the mapped data into the data calculation formula for calculation and processing, and to obtain the processed data linked list.

7. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the computer cluster data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the computer cluster data processing method as described in any one of claims 1 to 5.

Images