Integer parallel computing method and device based on distributed storage and computer equipment

Abstract

The invention belongs to the field of high-performance computing, and relates to an integer parallel computing method and device based on distributed storage and computer equipment, and the method comprises the steps of collecting real-time resource indexes, dynamically identifying fault nodes, triggering task migration, and performing data verification and hard disk fault detection. The weight value of each node is calculated, the nodes are arranged according to the descending order of the weight values, and the nodes with high load capacity are selected to distribute tasks; dynamically distributing a data generation task to a computing node, executing parallel computing, and performing distributed storage on a result; obtaining an operand, converting the operand into a first-order tensor form of a basic operand, serializing tensor data, and sending the serialized tensor data to a parallel computing layer; distributing a search task to a computing node, retrieving storage data in parallel, reading effective data from a storage layer, and combining search results into a partial sum; and summarizing and then outputting. The system has dynamic resource management and fault-tolerant capabilities, and can realize efficient task allocation and load balancing.

Description

Technical Field

[0001] This invention relates to the field of high-performance computing technology, and in particular to integer parallel computing methods, apparatus and computer equipment based on distributed storage. Background Technology

[0002] With the rapid development of cryptography, blockchain security protocols, and other fields, the demand for efficient computation of ultra-large-scale integers (such as 1024 bits and above) is becoming increasingly urgent. Current large integer computation technologies exhibit three typical implementation schemes and their limitations: Real-time computing bottlenecks: Traditional FFT / Karatsuba algorithms have high time complexity in ultra-large-scale integer scenarios; GPU heterogeneous acceleration: Relying on the CUDA architecture to achieve SIMD parallelization, while improving the efficiency of small-to-medium-scale computations, it is limited by GPU memory capacity and PCIe bus bandwidth, resulting in poor scalability and high hardware costs; Traditional distributed frameworks: Suffering from low task scheduling efficiency and high storage access latency. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides an integer parallel computing method based on distributed storage, employing the following technical solution, including the following steps: Collect real-time resource metrics, including CPU utilization, memory utilization, and network latency metrics for each computing node. Based on these real-time resource metrics, dynamically identify faulty nodes, trigger task migration, and perform data verification and hard disk fault detection. Based on the aforementioned real-time resource metrics, the weight value of each node is calculated, and the nodes are arranged in descending order of weight value, with priority given to assigning tasks to nodes with strong load capacity. The data generation task is dynamically allocated to computing nodes to perform parallel computing, and the results of the parallel computing are distributed and managed with replicas. Obtain the decimal or hexadecimal operands A and B input by the user, convert operands A and B into the first-order tensor form of the basic operand b, and serialize the tensor data before sending it to the parallel computing layer. The search task is assigned to computing nodes, stored data is retrieved in parallel, search conditions are verified, valid data is read from the storage layer, and the search results are combined into partial sums. Sum the partial sums of all compute nodes to obtain the final integer result. Convert the result in internal tensor form to decimal or hexadecimal format and output the large integer data in decimal or hexadecimal format.

[0004] Preferably, the steps of collecting real-time resource indicators, including CPU utilization, memory utilization, and network latency indicators of each computing node, and dynamically identifying faulty nodes, triggering task migration, and performing data verification and hard disk fault detection based on these real-time resource indicators, specifically include: The real-time resource metrics are collected, including CPU utilization, memory utilization, and network latency metrics for each computing node. Based on the real-time resource metrics, faulty nodes are dynamically identified, and task migration is triggered. Regularly verify the integrity and accuracy of data in the distributed storage layer, and perform data verification and hard drive failure detection on the distributed storage layer.

[0005] Preferably, the step of comprehensively analyzing the real-time resource indicators, calculating the weight value of each node, arranging the nodes in descending order of their weight values, and prioritizing the allocation of tasks to nodes with strong load capacity specifically includes: Based on the aforementioned real-time resource metrics, calculate the weight value for each node; Arrange the nodes in descending order of their weight values; Distribute tasks evenly across high-weight nodes to avoid local overload.

[0006] Preferably, the steps of dynamically allocating data generation tasks to computing nodes, performing parallel computing, and distributing and managing the results of parallel computing specifically include: Dynamically allocate data generation tasks to computing nodes to achieve parallel pre-computation; Compute nodes execute parallel pre-computation tasks and generate intermediate results; Intermediate results are stored in a distributed file system, supporting fast retrieval and replica management.

[0007] Preferably, the steps of obtaining the decimal or hexadecimal operands A and B input by the user, converting operands A and B into a first-order tensor form of the basic operand b, and serializing the tensor data before sending it to the parallel computing layer specifically include: Obtain the decimal or hexadecimal operands A and B input by the user, and parse operands A and B into a standardized format; Convert operands A and B into first-order tensor forms of the underlying operand b; Tensor data is serialized and sent to the parallel computing layer to ensure data integrity.

[0008] Preferably, the steps of allocating the search task to computing nodes, retrieving stored data in parallel, verifying search conditions, reading valid data from the storage layer, and combining the search results into a partial sum specifically include: The search task is assigned to computing nodes to retrieve stored data in parallel. Verify the search criteria and read valid data from the storage layer; Combine search results into partial sums to prepare for aggregation.

[0009] Preferably, the step of summing the partial sums of all computation nodes to obtain the final integer result, converting the result in internal tensor form to decimal or hexadecimal format, and outputting the large integer data in decimal or hexadecimal format specifically includes: Summarize the partial results from all computation nodes to obtain the final large integer; Convert the result in internal tensor form to decimal or hexadecimal format; The client interface layer outputs large integer data in decimal or hexadecimal format via a web interface or command line.

[0010] To address the aforementioned technical problems, the present invention also provides an integer parallel computing device based on distributed storage, employing the technical solution described below, including: The identification module is used to collect real-time resource indicators, including CPU utilization, memory utilization and network latency of each computing node. Based on the real-time resource indicators, the module dynamically identifies faulty nodes, triggers task migration, and performs data verification and hard disk fault detection. The allocation module is used to comprehensively analyze the real-time resource indicators, calculate the weight value of each node, arrange the nodes in descending order of weight value, and prioritize the allocation of tasks to nodes with strong load capacity. The storage module is used to dynamically allocate data generation tasks to computing nodes, perform parallel computing, and distribute and manage the results of parallel computing. The decomposition module is used to obtain the decimal or hexadecimal operands A and B input by the user, convert operands A and B into the first-order tensor form of the basic operand b, and serialize the tensor data before sending it to the parallel computing layer. The combination module is used to allocate search tasks to computing nodes, retrieve stored data in parallel, verify search conditions, read valid data from the storage layer, and combine search results into partial sums. The output module is used to summarize the partial sums of all computation nodes to obtain the final integer result, convert the result in internal tensor form to decimal or hexadecimal format, and output large integer data in decimal or hexadecimal format.

[0011] To address the aforementioned technical problems, the present invention also provides a computer device that employs the technical solution described below, comprising a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the aforementioned integer parallel computing method based on distributed storage.

[0012] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium, which employs the technical solution described below. The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the aforementioned integer parallel computing method based on distributed storage.

[0013] Compared with the prior art, the present invention has the following main advantages: Firstly, it possesses powerful dynamic resource management and fault tolerance capabilities: by collecting real-time resource indicators, it can accurately identify faulty nodes and trigger task migration, while simultaneously performing data verification and hard disk fault detection, ensuring stable operation of the system in complex environments and avoiding overall computational interruption due to individual node failures, thus greatly improving the reliability and availability of the system. Secondly, it can achieve efficient task allocation and load balancing: it calculates the node weight value based on real-time resource indicators, sorts them in descending order, and prioritizes the allocation of tasks to nodes with strong load capacity, fully tapping the computing potential of each node, making reasonable use of resources, effectively shortening computing time, and improving overall computing efficiency. Third, it supports flexible data processing and parallel computing: it converts user-input operands into first-order tensors, serializes them, and sends them to the parallel computing layer, dynamically allocates data generation tasks, realizes parallel computing and distributed storage, and can quickly process large-scale integer computing tasks. Fourth, it provides convenient result output and format conversion: after parallel retrieval of stored data and combination of partial sums to obtain the final integer result, it can be flexibly converted into decimal or hexadecimal format output to meet the needs of different users for result format, thereby improving the versatility and practicality of the method. Attached Figure Description

[0014] To more clearly illustrate the solutions in this invention, the accompanying drawings used in the description of the embodiments of this invention will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 This is a flowchart of an embodiment of the integer parallel computing method based on distributed storage of the present invention; Figure 2 This is an exemplary system architecture diagram in which the present invention can be applied; Figure 3 This is a schematic diagram of an embodiment of the integer parallel computing device based on distributed storage of the present invention; Figure 4 This is a schematic diagram of the structure of an embodiment of the computer device of the present invention. Detailed Implementation

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects and not to describe a particular order.

[0017] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0019] It should be noted that the integer parallel computing method based on distributed storage provided in the embodiments of the present invention is generally executed by a server / terminal device, and correspondingly, the integer parallel computing device based on distributed storage is generally set in the server / terminal device.

[0020] It should be understood that the number of terminal devices, networks, and servers is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be used.

[0021] Example 1 Please refer to Figure 1 The diagram illustrates a flowchart of an embodiment of the integer parallel computing method based on distributed storage according to the present invention. The integer parallel computing method based on distributed storage includes the following steps: Step S1: Collect real-time resource indicators, including CPU utilization, memory utilization, and network latency of each computing node. Based on the real-time resource indicators, dynamically identify faulty nodes, trigger task migration, and perform data verification and hard disk fault detection.

[0022] In this embodiment, the electronic device (e.g., a server / terminal device) on which the integer parallel computing method based on distributed storage runs can receive integer parallel computing requests based on distributed storage via wired or wireless connections. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAXX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods.

[0023] In this embodiment, step S1 specifically includes the following steps: S11 collects real-time resource metrics, including CPU utilization, memory utilization, and network latency metrics for each computing node.

[0024] A lightweight monitoring agent (such as Prometheus or a custom collection script) is deployed on each compute node to collect system metrics once per second. The metric data is transmitted to the scheduling management layer in real time via HTTP API or message queue (such as Kafka).

[0025] The collected metrics include: CPU utilization (%), obtained by reading the / proc / stat file or using the psutil library; memory utilization (%), obtained by parsing / proc / meminfo or system calls; and network latency (milliseconds), measured by ICMPping or TCP handshake time to determine inter-node communication latency.

[0026] As the core control unit, the scheduling management layer undertakes key responsibilities such as task allocation for computing nodes, aggregation of computing results, node health monitoring module, and fault handling. It is further composed of a computing node resource acquisition module, a data verification module, a dynamic task allocation system module, a fault migration module, and aggregation operators. All components work together to ensure the efficient and stable operation of the entire computing process.

[0027] The purpose of step S11 is to continuously collect key indicators such as CPU utilization, memory utilization, and network latency of each computing node, provide data support for dynamic task allocation, and avoid node overload.

[0028] S12 dynamically identifies faulty nodes based on real-time resource metrics and triggers task migration.

[0029] A multi-threshold judgment logic is set up: if a node's CPU utilization exceeds 90% for 5 consecutive seconds, its memory utilization exceeds 85% for 5 consecutive seconds, or its network latency exceeds 50 milliseconds for 5 consecutive seconds, then it is marked as a faulty node. A sliding time window algorithm can be used, and a state machine can be used to manage the node's health status. Faulty node information is stored in a distributed database (such as Redis) for the task allocation module to query.

[0030] The purpose of step S12 is to dynamically identify faulty nodes based on resource indicators, trigger task migration, and ensure computational continuity.

[0031] S13, periodically verify the integrity and accuracy of data in the distributed storage layer, and perform data verification and hard disk failure detection on the distributed storage layer.

[0032] The distributed storage layer provides data storage support and is a distributed file system composed of mobile hard drive arrays. It is mainly used to store the calculation results generated by computing nodes in the data generation stage, and to provide data sources for the search operations of the parallel computing layer in the data calculation stage. It adopts a distributed architecture design, which can effectively improve the reliability and access efficiency of data storage.

[0033] The scheduling management layer randomly reads 5% of the data from the distributed storage layer every 5 minutes (selecting data blocks via a random number generator) and performs two levels of verification. First, path verification: compare whether the data storage path (such as directory structure) is consistent with the metadata record.

[0034] Secondly, content verification: The read data is compared with the hash value of the original calculation result (such as SHA-256). If the verification fails, the hard drive is marked as a faulty hard drive. If the number of healthy hard drives is less than 1, the failover process is automatically triggered. Technical measures include distributed locks (such as Zookeeper) to avoid concurrent access conflicts.

[0035] The purpose of step S13 is to periodically verify the integrity and accuracy of the data in the distributed storage layer to prevent data loss due to hard drive failure.

[0036] Step S1 is fundamental to the stable operation of the system. By monitoring the status of computing nodes and storage devices in real time, the reliable execution of computing tasks is ensured. High availability is achieved in this embodiment by integrating multi-dimensional resource metrics and an automatic fault detection mechanism.

[0037] Step S2: Based on real-time resource metrics, calculate the weight value of each node, arrange the nodes in descending order of weight value, and prioritize assigning tasks to nodes with strong load capacity.

[0038] In this embodiment, step S2 specifically includes the following steps: S21 calculates the weight value of each node by integrating real-time resource metrics.

[0039] The weighting function is defined as: ,in: Let be the weight value of node i. This indicates CPU availability. This indicates memory availability. This represents network latency (normalized to the range of 0-1). This indicates adjustable weighting coefficients (e.g., default values ​​of 0.5, 0.3, and 0.2), which can be determined through experimental optimization.

[0040] Use numerical computing libraries such as NumPy for real-time computation.

[0041] The purpose of step S21 is to calculate the weight value of each node by combining CPU utilization, memory utilization and network latency indicators, and use it as the basis for task allocation.

[0042] S22, arrange the nodes in descending order of their weight values.

[0043] The quicksort algorithm is used to sort the list of nodes, with a time complexity of O(n log n). The sorting result is cached in memory and updated every 5 seconds to reduce scheduling overhead.

[0044] The purpose of step S22 is to sort the nodes in descending order of weight and prioritize assigning tasks to nodes with strong load capacity.

[0045] S23 distributes tasks evenly across high-weight nodes to avoid local overload.

[0046] Based on task size (e.g., computational cost estimate) and remaining node resources, a greedy algorithm allocates tasks to the nodes ranked higher in the timeline. It ensures that the load on each node does not exceed 80% of its maximum processing capacity (achieved through a resource reservation model). Task allocation instructions are sent to the compute nodes via RPC (e.g., gRPC).

[0047] The purpose of step S23 is to distribute tasks evenly across high-weight nodes to avoid local overload.

[0048] Step S2 achieves intelligent optimization of task allocation by quantifying node load capacity. By introducing a weighting function and sorting mechanism, load balancing and efficient resource utilization are ensured.

[0049] Step S3: Dynamically allocate data generation tasks to computing nodes, perform parallel computing, and distribute and manage the results of parallel computing.

[0050] In this embodiment, step S3 specifically includes the following steps: S31 dynamically allocates data generation tasks to computing nodes to achieve parallel pre-computation.

[0051] The task format is ,in and For parameters (e.g.) The scheduling management layer groups and distributes tasks according to the method in step S2. The task descriptor contains parameter ranges and storage path metadata.

[0052] S32, the computing node executes parallel pre-computation tasks to generate intermediate results.

[0053] The computation operator is based on a fast algorithm (such as optimized modular multiplication). For example, if the system is used for multiplication, then The computation process is accelerated using SIMD instructions (such as AVX).

[0054] S33 stores intermediate results in a distributed file system, supporting fast retrieval and replica management.

[0055] Storage employs a predefined strategy: Directory structure: of high The bit is used as the directory name ( (This refers to system parameters, such as 16-bit).

[0056] Filename: low The bit is used as the file name ( ).

[0057] Data format: Each record occupies bytes (e.g., 16 bytes), formatted as A 3-replica strategy (such as HDFS's replication mechanism) can be used to replicate data to different hard drives. The storage layer is implemented based on Ceph or a custom distributed file system.

[0058] Step S3 pre-calculates and stores intermediate results to provide data support for subsequent calculations. By combining distributed storage and replication strategies, data reliability and access efficiency are improved.

[0059] Step S4: Obtain the decimal or hexadecimal operands A and B input by the user, convert operands A and B into the first-order tensor form of the basic operand b, and serialize the tensor data before sending it to the parallel computing layer.

[0060] In this embodiment, step S4 specifically includes the following steps: S41: Obtain the decimal or hexadecimal operands A and B input by the user, and parse operands A and B into a standardized format.

[0061] The client interface layer provides a REST API or command-line tools to receive operands A and B. The parser uses a large integer library (such as GMP) to handle arbitrary precision numbers and eliminate format ambiguity.

[0062] The client interface layer serves as a bridge between the system and the user. It is responsible for receiving operands input by the user and displaying the final calculation results. It also performs operand format conversion to meet the system's internal calculation requirements. This layer consists of an input / output layer and a conversion operator module, providing a convenient user experience through a simple and intuitive interactive method.

[0063] S42 converts operands A and B into the first-order tensor form of the underlying operand b.

[0064] The conversion algorithm, based on Qin Jiushao's algorithm or iterative division, represents large integers as follows: ,in: Represents the input integer, Indicates the basic operand (system parameter, which can be a power of 2, such as...). ), Let be the tensor coefficients, satisfying , Let the length be the tensor. . The transformed tensor is denoted as Similarly, B is processed to obtain .

[0065] Using an optimized large number division algorithm can reduce computational overhead.

[0066] S43 serializes the tensor data and sends it to the parallel computing layer to ensure data integrity.

[0067] The parallel computing layer is the computation execution unit of the system. It is mainly responsible for receiving the computation tasks assigned by the scheduling management layer and performing corresponding computation operations according to different computation stages of the system. This includes writing the computation results to a specified area in the distributed storage layer, or searching for the required results from the distributed storage layer and performing combined computations. Finally, it feeds back the computation results to the scheduling management layer or the client interface layer. It is a key link in realizing parallel computing of large integers.

[0068] Serialization can be performed using Protocol Buffers or Avro to compress data size. Data can then be sent to compute nodes via zero-copy network transmission (such as RDMA) to reduce latency.

[0069] Steps S3 and S4 of the entire process include the data generation and data calculation stages.

[0070] Data generation: Based on the basic operand b, for example b=16, generate each record of the task: the result of the operation between each element of [1,2,3,...,15] and each element of [1,2,3,...,15]. Calculation process: Convert operands A and B into a first-order tensor of the base operand b. Search for the generated data based on the elements of the tensor to complete the operation on operands A and B.

[0071] Step S5: Assign the search task to the computing nodes, retrieve the stored data in parallel, verify the search conditions, read the valid data from the storage layer, and combine the search results into a partial sum.

[0072] In this embodiment, step S5 specifically includes the following steps: S51 distributes search tasks to computing nodes to retrieve stored data in parallel.

[0073] The search task is The scheduling management layer issues tasks according to a dynamic allocation strategy. Task parameters are generated from tensor components converted by the client.

[0074] S52, verify the search criteria and read valid data from the storage layer.

[0075] The search operator first checks the conditions. (Example conditions). If not met, return 0 directly. Otherwise, according to... Bit decomposition and location storage file: Read directory (high) (bit) and file (low) The first in the position) byte( Depend on calculate).

[0076] Validate data header and If a match is found, extract the value. Using memory-mapped files (mmap) speeds up reading and reduces I / O latency.

[0077] S53 combines search results into partial sums to prepare for aggregation.

[0078] Combinatorial operators include: basis operations and summation operations, etc.

[0079] Basis operation: For each search result Perform a left shift operation, i.e. (Equivalent to a binary left shift).

[0080] Summation operation: for all Summing yields a partial result. Parallel reduction algorithms (such as OpenMP) are used to perform the combination locally on the compute nodes.

[0081] Step S5 quickly retrieves pre-stored data using search operators and combines some results. Tensor search and pipelined computation are used to reduce real-time computation overhead.

[0082] Step S6: Sum the partial sums of all computation nodes to obtain the final integer result. Convert the result in internal tensor form to decimal or hexadecimal format and output the large integer data in decimal or hexadecimal format.

[0083] In this embodiment, step S6 specifically includes the following steps: S61, sum the partial results of all computation nodes to obtain the final large integer.

[0084] Aggregate operators perform global summation: ,in To determine the number of nodes, a distributed summation algorithm (such as the Reduce phase of MapReduce) is employed, and network overhead is reduced through a tree-structured communication model (such as MPI_Allreduce).

[0085] S62 converts the result in internal tensor form to decimal or hexadecimal format.

[0086] Inverse transformation algorithms, based on Horner's rule or iterative multiplication, restore tensor forms to large integers: ,in These are the coefficients after aggregation. Use a large number library (such as BN library) for efficient conversion processing.

[0087] S63, the client interface layer outputs large integer data in decimal or hexadecimal format through a web interface or command line.

[0088] It supports formatting enhancements (such as grouping numbers) and logging to ensure a better user experience.

[0089] Figure 2 This is an exemplary system architecture diagram in which the present invention can be applied. For example... Figure 2 As shown, the present invention can be applied to an exemplary system architecture, which includes four modules: a distributed storage layer, a parallel computing layer, and a client interface layer.

[0090] During the data generation phase, the scheduling management layer first obtains system information by reading the system parameter information module, specifically including weighting coefficients ( The system first obtains the basic operand 'b', the number of compute nodes and their IP information, then acquires the node status through the compute node resource acquisition module, and finally uses the dynamic task allocation system to generate tasks from the data. ( , )according to The data is distributed to the parallel computing layer, which performs the calculations using the computation operators and writes the results to the distributed storage layer in a predefined manner. Meanwhile, the scheduling management layer monitors the stored data in real time through the data verification module.

[0091] During the data computation phase, the client interface layer receives decimal / hexadecimal operands A and B input by the user. First, the conversion operator module converts operands A and B into a first-order tensor of system parameter b. Second, the scheduling management layer obtains the computing node status through the computing node resource acquisition module. Then, the dynamic task allocation system module distributes the search task to the parallel computing layer. The search operator module of the parallel computing layer retrieves the search results from the distributed storage layer according to the search task. Next, the data combination operator combines the search results. Finally, the aggregation operator of the scheduling management layer aggregates all the results from the parallel computing layer. Finally, the computation result is displayed to the user through the client interface layer.

[0092] In practice, the tasks of the scheduling management layer include: collecting computing node resources, performing dynamic task allocation, data verification, fault migration, and aggregation operators.

[0093] Collecting computing node resources refers to monitoring the operational status of each computing node in real time. Specific monitoring indicators include CPU utilization, memory utilization, and network latency (communication latency). By collecting and analyzing these indicators in real time, the system can provide node resource information to the dynamic task allocation system, enabling it to allocate computing tasks rationally. Furthermore, it can assess the health status of computing nodes. If a computing node's CPU utilization exceeds 90% for 5 consecutive seconds, its memory utilization exceeds 85% for 5 consecutive seconds, or its communication latency exceeds 50 milliseconds for 5 consecutive seconds, that computing node is marked as a faulty node.

[0094] Dynamic task allocation refers to the process of calculating the weight of each computing node based on information such as CPU utilization, memory usage, and network throughput obtained through methods for collecting computing node resources. The weight function is expressed as follows: ( After calculating the weight of the computing nodes according to the expression, the computing nodes are first sorted from high to low according to the weight. Then, according to the task size and the remaining resources of the node, the task is assigned to the computing nodes with higher ranking in sequence to ensure that the task load of each computing node does not exceed 80% of its maximum processing capacity.

[0095] Data verification is used to ensure the accuracy and reliability of data in the distributed storage layer. The specific operation process is as follows: Every 5 minutes, the scheduling management layer randomly reads 5% of the generated data from the distributed storage layer for verification. First, the specified content of the read data is compared with the corresponding directory and file information to check whether the data storage path and identifier are correct. Second, the specified content of the read data is compared with the calculation result when the data was initially generated to verify the accuracy of the data content. If either of the above two comparisons fails, the hard drive storing the data is marked as a faulty hard drive. If the number of healthy hard drives in the system is less than 1, a fault migration process is triggered.

[0096] Failover is primarily used to handle two scenarios: compute node failure and storage disk failure. When the method for collecting computing node resources detects a faulty node, the Checkpoint-Restart mode is used to migrate computing tasks. That is, by using the previously saved checkpoint data, the unfinished computing tasks on the faulty node are restarted and executed on other healthy nodes to ensure the continuity of computing tasks.

[0097] When the data verification method detects a faulty hard drive and the number of healthy hard drives is less than 1, the hard drive failure migration process is initiated. Specifically, a new hard drive is added, and the data stored in the healthy hard drive is completely copied to the new hard drive to restore the system's storage capacity and data reliability, ensuring the normal operation of subsequent data access and calculation operations.

[0098] Aggregation operators refer to operators that, during the data computation phase, occur when all computing nodes in the system have completed their assigned computational tasks and returned the results. The aggregation operator performs a summation operation, that is... The results are then output to the client interface layer so that the client interface layer can display the calculation results to the user.

[0099] The distributed storage layer, composed of a distributed file system composed of arrays of portable hard drives, primarily performs the following tasks: It employs a distributed storage architecture to distribute data across multiple portable hard drives, achieving unified data management and access through the distributed file system's management mechanism. During data storage, combined with a 3-replica strategy from the data generation phase, the computation results generated by the computing nodes are stored as three copies on different portable hard drives. This multi-replica storage method effectively improves data storage reliability. Simultaneously, the distributed storage architecture enables parallel data access; multiple computing nodes can simultaneously read or write data from different hard drives, significantly improving data access efficiency and providing strong support for the system's parallel computing.

[0100] The client interface layer primarily performs tasks including data input / output and transformation operators. The input / output layer performs data input / output tasks.

[0101] Data input / output refers to the data interaction between the system and the user. Specific functions include receiving two decimal / hexadecimal operands A and B from the user. These two operands are the basic data for subsequent integer operations. After the calculation process is completed, the system is responsible for displaying the final calculation result of the basic numbers to the user in an intuitive and clear form, so that the user can easily obtain the calculation result.

[0102] A transformation operator is a function that converts operands A and B received through data input and output into the form of a first-order tensor b, denoted as b. , ,in , , ,and , The transformation results are then sent to the parallel computing layer.

[0103] The parallel computing layer refers to the layer that receives tasks assigned by the scheduling management layer according to different stages of the system (data generation stage or data computation stage) and performs corresponding computation or data search operations. The specific process is as follows: During the data generation phase, the computing nodes in the parallel computing layer receive computing tasks assigned by the scheduling management layer. ,in , After the computing nodes complete the calculations according to the task requirements, they use a 3-replica strategy to write the calculation results into a designated area of ​​the distributed storage layer in a predefined manner.

[0104] The predefined method refers to using the first operand in the computation task. of high The low-order bits are used as storage directories. The filename is defined by the number of bits used, and each calculation result occupies [a certain number of bits]. Each record is in the form of bytes. ,in , .

[0105] During the data computation phase, the computing nodes receive computing tasks assigned by the scheduling management layer. The computing nodes execute the search operator in parallel to obtain... The specific process is as follows: First, determine whether the task parameters meet the conditions. This condition is denoted as C.

[0106] If condition C is not met, then The corresponding result is 0. Otherwise, perform the following operation to retrieve data from the distributed storage layer. high The directory corresponding to the bit, low Read the file corresponding to the first bit. The bytes read are denoted as: If the 1st to The value corresponding to the bit and Consistent, the first +1~2 Position and If they match, then the value corresponding to the remaining bits is... The value; After the search is completed, the computing nodes perform combined calculations on the search results using data combination operators. This combined calculation process includes basis operations and summation operations in sequence: Basis operations refer to the operations performed by the search operator. The result is then left-shifted, i.e.: ; The summation operation refers to summing the results of the radix operations according to j, and the result is denoted as j. ,but .

[0107] Finally, the computing node returns the result of the combined computation to The scheduling management layer is configured to perform subsequent aggregation operations.

[0108] The beneficial effects of implementing this embodiment are: Firstly, it possesses powerful dynamic resource management and fault tolerance capabilities: by collecting real-time resource indicators, it can accurately identify faulty nodes and trigger task migration, while simultaneously performing data verification and hard disk fault detection, ensuring stable operation of the system in complex environments and avoiding overall computational interruption due to individual node failures, thus greatly improving the reliability and availability of the system. Secondly, it can achieve efficient task allocation and load balancing: it calculates the node weight value based on real-time resource indicators, sorts them in descending order, and prioritizes the allocation of tasks to nodes with strong load capacity, fully tapping the computing potential of each node, making reasonable use of resources, effectively shortening computing time, and improving overall computing efficiency. Third, it supports flexible data processing and parallel computing: it converts user-input operands into first-order tensors, serializes them, and sends them to the parallel computing layer, dynamically allocates data generation tasks, realizes parallel computing and distributed storage, and can quickly process large-scale integer computing tasks. Fourth, it provides convenient result output and format conversion: after parallel retrieval of stored data and combination of partial sums to obtain the final integer result, it can be flexibly converted into decimal or hexadecimal format output to meet the needs of different users for result format, thereby improving the versatility and practicality of the method.

[0109] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0111] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0112] Example 2 Further reference Figure 3 As a response to the above Figure 1The present invention provides an embodiment of an integer parallel computing device based on distributed storage, which is implemented in accordance with the method shown. Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0113] like Figure 3 As shown, the integer parallel computing device 70 based on distributed storage in this embodiment includes: an identification module 71, an allocation module 72, a storage module 73, a decomposition module 74, a combination module 75, and an output module 76. Wherein: The identification module 71 is used to collect real-time resource indicators, including CPU utilization, memory utilization and network latency of each computing node. Based on the real-time resource indicators, it dynamically identifies faulty nodes, triggers task migration, and performs data verification and hard disk fault detection. The allocation module 72 is used to comprehensively analyze real-time resource indicators, calculate the weight value of each node, arrange the nodes in descending order of weight value, and prioritize the allocation of tasks to nodes with strong load capacity. Storage module 73 is used to dynamically allocate data generation tasks to computing nodes, perform parallel computing, and perform distributed storage and replica management of parallel computing results; The decomposition module 74 is used to obtain the decimal or hexadecimal operands A and B input by the user, convert operands A and B into the first-order tensor form of the basic operand b, and serialize the tensor data before sending it to the parallel computing layer. The combination module 75 is used to allocate search tasks to computing nodes, retrieve stored data in parallel, verify search conditions, read valid data from the storage layer, and combine search results into partial sums. Output module 76 is used to summarize the partial sums of all computation nodes to obtain the final integer result, convert the result in internal tensor form to decimal or hexadecimal format, and output large integer data in decimal or hexadecimal format.

[0114] The beneficial effects of implementing this embodiment are: Firstly, it possesses powerful dynamic resource management and fault tolerance capabilities: by collecting real-time resource indicators, it can accurately identify faulty nodes and trigger task migration, while simultaneously performing data verification and hard disk fault detection, ensuring stable operation of the system in complex environments and avoiding overall computational interruption due to individual node failures, thus greatly improving the reliability and availability of the system. Secondly, it can achieve efficient task allocation and load balancing: it calculates the node weight value based on real-time resource indicators, sorts them in descending order, and prioritizes the allocation of tasks to nodes with strong load capacity, fully tapping the computing potential of each node, making reasonable use of resources, effectively shortening computing time, and improving overall computing efficiency. Third, it supports flexible data processing and parallel computing: it converts user-input operands into first-order tensors, serializes them, and sends them to the parallel computing layer, dynamically allocates data generation tasks, realizes parallel computing and distributed storage, and can quickly process large-scale integer computing tasks. Fourth, it provides convenient result output and format conversion: after parallel retrieval of stored data and combination of partial sums to obtain the final integer result, it can be flexibly converted into decimal or hexadecimal format output to meet the needs of different users for result format, thereby improving the versatility and practicality of the method.

[0115] Example 3 To address the aforementioned technical problems, embodiments of the present invention also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0116] The aforementioned computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that only the computer device 8 with components 81, 82, and 83 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0117] The aforementioned computer devices can be desktop computers, laptops, handheld computers, and cloud servers, among other computing devices. These devices can facilitate human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0118] The aforementioned memory 81 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the aforementioned memory 81 may be an internal storage unit of the aforementioned computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the aforementioned memory 81 may also be an external storage device of the aforementioned computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the aforementioned memory 81 may also include both the internal storage unit and its external storage device of the aforementioned computer device 8. In this embodiment, the aforementioned memory 81 is typically used to store the operating system and various application software installed on the aforementioned computer device 8, such as computer-readable instructions based on a distributed storage-based integer parallel computing method. In addition, the aforementioned memory 81 can also be used to temporarily store various types of data that have been output or will be output.

[0119] In some embodiments, the processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to execute computer-readable instructions stored in the memory 81 or to process data, for example, to execute computer-readable instructions of the distributed memory-based integer parallel computing method.

[0120] The network interface 83 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 8 and other electronic devices.

[0121] The beneficial effects of implementing this embodiment are: Firstly, it possesses powerful dynamic resource management and fault tolerance capabilities: by collecting real-time resource indicators, it can accurately identify faulty nodes and trigger task migration, while simultaneously performing data verification and hard disk fault detection, ensuring stable operation of the system in complex environments and avoiding overall computational interruption due to individual node failures, thus greatly improving the reliability and availability of the system. Secondly, it can achieve efficient task allocation and load balancing: it calculates the node weight value based on real-time resource indicators, sorts them in descending order, and prioritizes the allocation of tasks to nodes with strong load capacity, fully tapping the computing potential of each node, making reasonable use of resources, effectively shortening computing time, and improving overall computing efficiency. Third, it supports flexible data processing and parallel computing: it converts user-input operands into first-order tensors, serializes them, and sends them to the parallel computing layer, dynamically allocates data generation tasks, realizes parallel computing and distributed storage, and can quickly process large-scale integer computing tasks. Fourth, it provides convenient result output and format conversion: after parallel retrieval of stored data and combination of partial sums to obtain the final integer result, it can be flexibly converted into decimal or hexadecimal format output to meet the needs of different users for result format, thereby improving the versatility and practicality of the method.

[0122] Example 4 The present invention also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the integer parallel computing method based on distributed storage as described above.

[0123] The beneficial effects of implementing this embodiment are: Firstly, it possesses powerful dynamic resource management and fault tolerance capabilities: by collecting real-time resource indicators, it can accurately identify faulty nodes and trigger task migration, while simultaneously performing data verification and hard disk fault detection, ensuring stable operation of the system in complex environments and avoiding overall computational interruption due to individual node failures, thus greatly improving the reliability and availability of the system. Secondly, it can achieve efficient task allocation and load balancing: it calculates the node weight value based on real-time resource indicators, sorts them in descending order, and prioritizes the allocation of tasks to nodes with strong load capacity, fully tapping the computing potential of each node, making reasonable use of resources, effectively shortening computing time, and improving overall computing efficiency. Third, it supports flexible data processing and parallel computing: it converts user-input operands into first-order tensors, serializes them, and sends them to the parallel computing layer, dynamically allocates data generation tasks, realizes parallel computing and distributed storage, and can quickly process large-scale integer computing tasks. Fourth, it provides convenient result output and format conversion: after parallel retrieval of stored data and combination of partial sums to obtain the final integer result, it can be flexibly converted into decimal or hexadecimal format output to meet the needs of different users for result format, thereby improving the versatility and practicality of the method.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0125] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.

Claims

1. A parallel integer computing method based on distributed storage, characterized in that, Includes the following steps: Collect real-time resource metrics, including CPU utilization, memory utilization, and network latency metrics for each computing node. Based on these real-time resource metrics, dynamically identify faulty nodes, trigger task migration, and perform data verification and hard disk fault detection. Based on the aforementioned real-time resource metrics, the weight value of each node is calculated, and the nodes are arranged in descending order of weight value, with priority given to assigning tasks to nodes with strong load capacity. The data generation task is dynamically allocated to computing nodes to perform parallel computing, and the results of the parallel computing are distributed and managed with replicas. Obtain the decimal or hexadecimal operands A and B input by the user, convert operands A and B into the first-order tensor form of the basic operand b, and serialize the tensor data before sending it to the parallel computing layer. The search task is assigned to computing nodes, stored data is retrieved in parallel, search conditions are verified, valid data is read from the storage layer, and the search results are combined into partial sums. Sum the partial sums of all compute nodes to obtain the final integer result. Convert the result in internal tensor form to decimal or hexadecimal format and output the large integer data in decimal or hexadecimal format.

2. The integer parallel computing method based on distributed storage according to claim 1, characterized in that, The steps of collecting real-time resource metrics, including CPU utilization, memory utilization, and network latency metrics for each computing node, and dynamically identifying faulty nodes, triggering task migration, and performing data verification and hard disk fault detection based on these real-time resource metrics, specifically include: The real-time resource metrics are collected, including CPU utilization, memory utilization, and network latency metrics for each computing node. Based on the real-time resource metrics, faulty nodes are dynamically identified, and task migration is triggered. Regularly verify the integrity and accuracy of data in the distributed storage layer, and perform data verification and hard drive failure detection on the distributed storage layer.

3. The integer parallel computing method based on distributed storage according to claim 1, characterized in that, The steps of comprehensively analyzing the real-time resource metrics, calculating the weight value of each node, arranging the nodes in descending order of their weight values, and prioritizing the allocation of tasks to nodes with strong load capacity specifically include: Based on the aforementioned real-time resource metrics, calculate the weight value for each node; Arrange the nodes in descending order of their weight values; Distribute tasks evenly across high-weight nodes to avoid local overload.

4. The integer parallel computing method based on distributed storage according to claim 1, characterized in that, The steps of dynamically allocating data generation tasks to computing nodes, performing parallel computing, and distributing and managing the results of parallel computing in a distributed manner include: Dynamically allocate data generation tasks to computing nodes to achieve parallel pre-computation; Compute nodes execute parallel pre-computation tasks and generate intermediate results; Intermediate results are stored in a distributed file system, supporting fast retrieval and replica management.

5. The integer parallel computing method based on distributed storage according to claim 1, characterized in that, The steps of obtaining the decimal or hexadecimal operands A and B input by the user, converting operands A and B into a first-order tensor form of the basic operand b, and serializing the tensor data before sending it to the parallel computing layer specifically include: Obtain the decimal or hexadecimal operands A and B input by the user, and parse operands A and B into a standardized format; Convert operands A and B into first-order tensor forms of the underlying operand b; Tensor data is serialized and sent to the parallel computing layer to ensure data integrity.

6. The integer parallel computing method based on distributed storage according to claim 1, characterized in that, The steps of allocating search tasks to computing nodes, retrieving stored data in parallel, verifying search conditions, reading valid data from the storage layer, and combining search results into a partial sum specifically include: The search task is assigned to computing nodes to retrieve stored data in parallel. Verify the search criteria and read valid data from the storage layer; Combine search results into partial sums to prepare for aggregation.

7. The integer parallel computing method based on distributed storage according to any one of claims 1 to 6, characterized in that, The steps of summing the partial sums of all computation nodes to obtain the final integer result, converting the result in internal tensor form to decimal or hexadecimal format, and outputting the large integer data in decimal or hexadecimal format specifically include: Summarize the partial results from all computation nodes to obtain the final large integer; Convert the result in internal tensor form to decimal or hexadecimal format; The client interface layer outputs large integer data in decimal or hexadecimal format via a web interface or command line.

8. An integer parallel computing device based on distributed storage, characterized in that, include: The identification module is used to collect real-time resource indicators, including CPU utilization, memory utilization and network latency of each computing node. Based on the real-time resource indicators, the module dynamically identifies faulty nodes, triggers task migration, and performs data verification and hard disk fault detection. The allocation module is used to comprehensively analyze the real-time resource indicators, calculate the weight value of each node, arrange the nodes in descending order of weight value, and prioritize the allocation of tasks to nodes with strong load capacity. The storage module is used to dynamically allocate data generation tasks to computing nodes, perform parallel computing, and distribute and manage the results of parallel computing. The decomposition module is used to obtain the decimal or hexadecimal operands A and B input by the user, convert operands A and B into the first-order tensor form of the basic operand b, and serialize the tensor data before sending it to the parallel computing layer. The combination module is used to allocate search tasks to computing nodes, retrieve stored data in parallel, verify search conditions, read valid data from the storage layer, and combine search results into partial sums. The output module is used to summarize the partial sums of all computation nodes to obtain the final integer result, convert the result in internal tensor form to decimal or hexadecimal format, and output large integer data in decimal or hexadecimal format.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the integer parallel computing method based on distributed storage as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the integer parallel computing method based on distributed storage as described in any one of claims 1 to 7.

Images