Distributed storage-based integer parallel computing method and device, and computer equipment

Abstract

The application belongs to the field of high-performance computing and relates to an integer parallel computing method and device based on distributed storage and a computer device. The method comprises the following steps: collecting real-time resource indexes, dynamically identifying fault nodes, triggering task migration, and performing data verification and hard disk fault detection; calculating the weight value of each node, arranging the nodes in descending order of the weight value, and selecting a node with strong load capacity to allocate tasks; dynamically allocating data generation tasks to the computing nodes, performing parallel computing, and performing distributed storage on the results; obtaining operands and converting them into the first-order tensor form of the base operands, serializing the tensor data, and sending the tensor data to the parallel computing layer; allocating search tasks to the computing nodes, performing parallel search on the stored data, reading valid data from the storage layer, and combining the search results into partial sums; and performing summarization and outputting. The application has dynamic resource management and fault tolerance capabilities and can realize efficient task allocation and load balancing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of high-performance computing, in particular to an integer parallel computing method and device based on distributed storage and a computer device. BACKGROUND

[0002] With the rapid development of cryptography, blockchain security protocols and other fields, the demand for efficient operation of super large-scale integers (such as 1024 bits or more) is increasingly urgent. There are three typical implementation schemes and their limitations in current large integer operation technology: real-time computing bottleneck: the time complexity of traditional FFT / Karatsuba algorithm is high in the context of super large-scale integers; GPU heterogeneous acceleration: relying on CUDA architecture to realize SIMD parallelization, although it can improve the efficiency of small-scale operation, but it is limited by the memory capacity and PCle bus bandwidth, and has poor scalability and high hardware cost; traditional distributed framework: there are problems such as low task scheduling efficiency, high storage access delay, etc. SUMMARY

[0003] In order to solve the above technical problems, the present application provides an integer parallel computing method based on distributed storage, which adopts the following technical scheme, including the following steps:

[0004] Collect real-time resource indicators, including CPU utilization, memory utilization and network delay indicators of each computing node, based on the real-time resource indicators, dynamically identify fault nodes, trigger task migration, and perform data verification and hard disk fault detection;

[0005] Synthesize the real-time resource indicators to calculate the weight value of each node, arrange the nodes in descending order of weight value, and preferentially select nodes with strong load capacity to allocate tasks;

[0006] Dynamically allocate data generation tasks to computing nodes, perform parallel computing, and distribute and manage the parallel computing results;

[0007] Get the decimal or hexadecimal operands A and B input by the user, convert the operands A and B into the first-order tensor form of the basic operand b, serialize the tensor data and send it to the parallel computing layer;

[0008] Distribute search tasks to computing nodes, perform parallel search on storage data, verify search conditions, read valid data from the storage layer, and combine search results into partial sums;

[0009] Sum up the partial sums of all computing nodes to obtain the final integer result, convert the internal tensor form of the result into decimal or hexadecimal format, and output the decimal or hexadecimal format large integer data.

[0010] Preferably, the step of collecting real-time resource indicators, the real-time resource indicators including CPU utilization, memory utilization and network delay indicators of each computing node, dynamically identifying a faulty node based on the real-time resource indicators, triggering task migration, and performing data verification and hard disk failure detection specifically comprises:

[0011] collecting the real-time resource indicators, the real-time resource indicators including CPU utilization, memory utilization and network delay indicators of each computing node;

[0012] dynamically identifying a faulty node based on the real-time resource indicators, and triggering task migration;

[0013] periodically verifying the integrity and accuracy of the data of the distributed storage layer, and performing data verification and hard disk failure detection on the distributed storage layer.

[0014] Preferably, the step of integrating the real-time resource indicators, calculating a weight value of each node, and arranging the nodes in descending order of weight value, and preferentially selecting a node with strong load capacity to allocate tasks specifically comprises:

[0015] integrating the real-time resource indicators, and calculating a weight value of each node;

[0016] arranging the nodes in descending order of weight value;

[0017] equally allocating tasks to high-weight nodes to avoid local overload.

[0018] Preferably, the step of dynamically allocating data generation tasks to computing nodes, performing parallel computing, and performing distributed storage and replica management of parallel computing results specifically comprises:

[0019] dynamically allocating data generation tasks to computing nodes to achieve parallel pre-computation;

[0020] computing nodes perform parallel pre-computation tasks to generate intermediate results;

[0021] storing the intermediate results to a distributed file system to support fast retrieval and perform replica management.

[0022] Preferably, the step of obtaining decimal or hexadecimal operands A and B input by a user, converting the operands A and B into a first-order tensor form of a basic operand b, and serializing the tensor data and sending it to a parallel computing layer specifically comprises:

[0023] obtaining decimal or hexadecimal operands A and B input by a user, and parsing the operands A and B into a standardized format;

[0024] converting the operands A and B into a first-order tensor form of a basic operand b;

[0025] The serialized tensor data is sent to the parallel computing layer to ensure data integrity.

[0026] Preferably, the step of distributing the search task to the computing nodes, retrieving the stored data in parallel, verifying the search condition, reading the valid data from the storage layer, and combining the search results into partial sums specifically includes:

[0027] Distributing the search task to the computing nodes and retrieving the stored data in parallel;

[0028] Verifying the search condition and reading the valid data from the storage layer;

[0029] Combining the search results into partial sums and preparing for aggregation.

[0030] Preferably, the step of aggregating the partial sums of all computing nodes to obtain a final integer result, converting the result in the form of an internal tensor into a decimal or hexadecimal format, and outputting the decimal or hexadecimal format large integer data specifically includes:

[0031] Aggregating the partial results of all computing nodes to obtain a final large integer;

[0032] Converting the result in the form of an internal tensor into a decimal or hexadecimal format;

[0033] The client interface layer outputs the decimal or hexadecimal format large integer data through a Web interface or a command line.

[0034] To solve the above technical problems, the application also provides an integer parallel computing device based on distributed storage, which adopts the following technical solutions and comprises:

[0035] An identification module is configured to collect real-time resource indicators, wherein the real-time resource indicators include CPU utilization, memory utilization, and network delay indicators of each computing node, and based on the real-time resource indicators, a faulty node is dynamically identified, task migration is triggered, data verification and hard disk fault detection are performed;

[0036] An allocation module is configured to comprehensively consider the real-time resource indicators, calculate the weight value of each node, arrange the nodes in descending order of weight value, and preferentially select nodes with strong load capacity to allocate tasks;

[0037] A storage module is configured to dynamically allocate data generation tasks to computing nodes, perform parallel computing, and perform distributed storage and replica management on the parallel computing results.

[0038] A decomposition module is configured to obtain decimal or hexadecimal operands A and B input by a user, convert the operands A and B into a first-order tensor form of a basic operand b, and send the serialized tensor data to a parallel computing layer.

[0039] A combination module is configured to allocate the search task to the computing nodes, retrieve the stored data in parallel, verify the search condition, read the valid data from the storage layer, and combine the partial sums into a final integer result.

[0040] An output module is configured to aggregate the partial sums of all the computing nodes to obtain the final integer result, convert the result in the form of an internal tensor into a decimal or hexadecimal format, and output the large integer data in the decimal or hexadecimal format.

[0041] To solve the above technical problems, the present application also provides a computer device, which adopts the technical scheme as described below, comprising a memory and a processor, the memory stores computer readable instructions, and the processor implements the steps of the above-mentioned integer parallel computing method based on distributed storage when executing the computer readable instructions.

[0042] To solve the above technical problems, the present application also provides a computer readable storage medium, which adopts the technical scheme as described below, the computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the above-mentioned integer parallel computing method based on distributed storage.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] Firstly, it has strong dynamic resource management and fault tolerance capability: by collecting real-time resource indicators, it can accurately identify faulty nodes and trigger task migration, while performing data verification and hard disk fault detection, ensuring stable operation of the system in complex environments, avoiding overall calculation interruption due to individual node failure, and greatly improving the reliability and availability of the system;

[0045] Secondly, it can realize efficient task allocation and load balancing: the weight values of the computing nodes are calculated based on real-time resource indicators, and the nodes with strong load capacity are selected in descending order to allocate tasks, fully tapping the computing potential of each node, making the resources be reasonably utilized, effectively shortening the calculation time and improving the overall calculation efficiency;

[0046] Thirdly, it can support flexible data processing and parallel computing: the user input operands are converted into first-order tensor form, serialized and sent to the parallel computing layer, and the data generation tasks are dynamically allocated to realize parallel computing and distributed storage, which can quickly process large-scale integer calculation tasks;

[0047] Fourthly, it can provide convenient result output and format conversion: the stored data is retrieved in parallel to combine partial sums, and after aggregating the final integer result, it can be flexibly converted into a decimal or hexadecimal format for output, meeting the needs of different users for the result format and improving the universality and practicality of the method. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the solutions in the present application, the following will briefly introduce the drawings needed in the description of the embodiments of the present application. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0049] Figure 1 is a flow chart of an embodiment of the parallel integer calculation method based on distributed storage of the present application;

[0050] Figure 2 is an exemplary system architecture diagram to which the present application can be applied;

[0051] Figure 3 is a structural schematic diagram of an embodiment of the parallel integer calculation device based on distributed storage of the present application;

[0052] Figure 4 is a structural schematic diagram of an embodiment of the computer device of the present application. DETAILED DESCRIPTION

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terminology used in the description of the specification of the application is only for the purpose of describing specific embodiments and is not intended to limit the present application; the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above description of drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the specification and claims of the present application or the above description of drawings are used to distinguish different objects, not to describe a particular order.

[0054] Reference herein to "embodiments" means that the particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] In order to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings.

[0056] It should be noted that the parallel integer calculation method based on distributed storage provided by the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the parallel integer calculation device based on distributed storage is generally arranged in the server / terminal device.

[0057] It should be understood that the number of terminal devices, networks and servers is only illustrative. Any number of terminal devices, networks and servers can be provided according to implementation needs.

[0058] Embodiment one

[0059] Referring to Figure 1 , a flowchart of one embodiment of the distributed storage-based integer parallel computing method of the present application is shown. The distributed storage-based integer parallel computing method comprises the following steps:

[0060] Step S1, collecting real-time resource indicators, the real-time resource indicators including CPU utilization, memory utilization and network delay indicators of each computing node, based on the real-time resource indicators, dynamically identifying a faulty node, triggering task migration, performing data verification and hard disk fault detection.

[0061] In this embodiment, the electronic device (such as a server / terminal device) on which the distributed storage-based integer parallel computing method runs can receive a distributed storage-based integer parallel computing request through a wired connection or a wireless connection. It should be noted that the wireless connection can include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAXX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection methods.

[0062] In this embodiment, step S1 specifically comprises the following steps:

[0063] S11, collecting real-time resource indicators, the real-time resource indicators including CPU utilization, memory utilization and network delay indicators of each computing node.

[0064] A lightweight monitoring agent (such as Prometheus or a custom collection script) is deployed on each computing node to collect system indicators at a frequency of once per second. The indicator data is transmitted in real time to the scheduling management layer through an HTTP API or a message queue (such as Kafka).

[0065] The collected indicators include: CPU utilization (%), obtained by reading / proc / stat file or using psutil library; memory utilization (%), obtained by parsing / proc / meminfo or system call; network delay (milliseconds), obtained by measuring inter-node communication delay through ICMP ping or TCP handshake time.

[0066] The scheduling management layer, as a core control unit, undertakes key responsibilities such as task allocation of computing nodes, aggregation of computing results, node health monitoring module and fault handling, and is further composed of a computing node resource acquisition module, a data verification module, a dynamic task allocation system module, a fault migration module and an aggregation operator, with the components working in coordination to ensure efficient and stable operation of the entire computing process.

[0067] The step S11 serves to continuously collect key indicators such as CPU utilization, memory utilization and network delay of each computing node, to provide data support for dynamic task allocation and avoid node overload.

[0068] S12, based on real-time resource indicators, dynamically identifies fault nodes and triggers task migration.

[0069] A multi-threshold judgment logic is set, and if the CPU utilization of a node exceeds 90% for 5 seconds, the memory utilization exceeds 85% for 5 seconds, or the network delay exceeds 50 milliseconds for 5 seconds, the node is marked as a fault node. A sliding time window algorithm can be used to manage the health status of the node through a state machine. The fault node information is stored in a distributed database (such as Redis) for the task allocation module to query.

[0070] The step S12 serves to dynamically identify fault nodes based on resource indicators, trigger task migration and ensure computing continuity.

[0071] S13, periodically verify the integrity and accuracy of the data in the distributed storage layer, and perform data verification and hard disk fault detection on the distributed storage layer.

[0072] The distributed storage layer provides data storage support and is a distributed file system composed of mobile hard disk arrays, mainly used for storing computing results generated by computing nodes in the data generation stage, and providing data sources for search operations of the parallel computing layer in the data computing stage. The distributed architecture design can effectively improve the reliability and access efficiency of data storage.

[0073] The scheduling management layer randomly reads 5% of the data in the distributed storage layer every 5 minutes (selects data blocks through a random number generator), and performs two-level verification:

[0074] First, path verification: compare whether the data storage path (such as directory structure) is consistent with the metadata record.

[0075] Second, content verification: compare the read data with the original computing result hash value (such as SHA-256). If the verification fails, the hard disk is marked as a fault hard disk. If the number of healthy hard disks is less than 1, the fault migration process is automatically triggered. Technical means include distributed locks (such as Zookeeper) to avoid concurrent access conflicts.

[0076] The step S13 is to periodically verify the integrity and accuracy of the data in the distributed storage layer, to prevent data loss due to hard disk failure.

[0077] The step S1 is the basis for stable operation of the system, by monitoring the status of the computing nodes and storage devices in real time, to ensure reliable execution of computing tasks. By integrating multi-dimensional resource indicators and automatic fault detection mechanisms, high availability is achieved in the embodiment.

[0078] In step S2, the real-time resource indicators are integrated to calculate the weight value of each node, and the nodes are arranged in descending order of weight value, and the nodes with strong load capacity are preferentially selected for task allocation.

[0079] In the embodiment, the step S2 specifically includes the steps of:

[0080] S21, integrating real-time resource indicators, calculating the weight value of each node.

[0081] The weight function is defined as: wherein: wi is the weight value of node i, represents the CPU availability rate, represents the memory availability rate, represents the network delay (normalized to the range of 0-1), represents the adjustable weight coefficient (such as the default values are 0.5, 0.3, 0.2), which can be determined by experiment optimization.

[0082] A numerical calculation library (such as NumPy) is used for real-time calculation.

[0083] The step S21 is to integrate CPU utilization, memory utilization and network delay indicators to calculate the weight value of each node as a basis for task allocation.

[0084] S22, arranging the nodes in descending order of weight value.

[0085] A quicksort algorithm is used to sort the node list, with a time complexity of O(n log n). The sorting result is cached in memory and updated every 5 seconds to reduce scheduling overhead.

[0086] The step S22 is to arrange the nodes in descending order of weight, and preferentially select the nodes with strong load capacity for task allocation.

[0087] S23, evenly distribute tasks to high-weight nodes to avoid local overload.

[0088] According to the task size (such as the calculation amount estimation) and the remaining resources of the nodes, a greedy algorithm is used to assign tasks to the nodes ranked first. Ensure that the load of each node does not exceed 80% of its maximum processing capacity (achieved through a resource reservation model). Task allocation instructions are issued to computing nodes through RPC (such as gRPC).

[0089] The role of step S23 is to evenly distribute tasks to high-weight nodes to avoid local overload.

[0090] Step S2 achieves intelligent optimization of task allocation by quantifying node load capacity. By introducing a weight function and sorting mechanism, load balancing and efficient resource utilization are ensured.

[0091] Step S3 dynamically assigns data generation tasks to computing nodes for parallel computing, and performs distributed storage and replica management of parallel computing results.

[0092] In this embodiment, step S3 specifically includes the following steps:

[0093] S31, dynamically assign data generation tasks to computing nodes to achieve parallel pre-computation.

[0094] The task form is , where and are parameters (for example ). The scheduling management layer issues task groups according to the method of step S2. The task descriptor contains parameter ranges and storage path metadata.

[0095] S32, the computing nodes execute parallel pre-computation tasks to generate intermediate results.

[0096] The calculation operator calculates based on a fast algorithm (such as an optimized multiplication modulo operation). For example, if the system is used for multiplication, then The calculation process is accelerated using SIMD instructions (such as AVX).

[0097] S33, store the intermediate results to the distributed file system to support fast retrieval and perform replica management.

[0098] The storage uses a predefined strategy:

[0099] Directory structure: use the high bit of as the directory name (the system parameter is 16 bits).

[0100] File name: use the low bit of as the file name (). ​

[0101] Data format: Each record occupies bytes (e.g. 16 bytes), in the format of . A 3-copy strategy (such as the replication mechanism of HDFS) can be used to replicate data to different hard disks. The storage layer is based on Ceph or a custom distributed file system implementation.

[0102] Step S3 precomputes and stores intermediate results to provide data support for subsequent calculations. By combining distributed storage and replication strategies, data reliability and access efficiency are improved.

[0103] Step S4, get the user input decimal or hexadecimal operand A, B, convert the operand A and B to the first-order tensor form of the base operand b, serialize the tensor data and send it to the parallel computing layer.

[0104] In this embodiment, step S4 specifically includes the steps of:

[0105] S41, get the user input decimal or hexadecimal operand A, B, and parse the operand A and B into a standardized format.

[0106] Through the client interface layer, REST API or command line tools are provided to receive operands A and B. The parser uses a large integer library (such as GMP) to handle arbitrary precision numbers and eliminate format ambiguity.

[0107] The client interface layer is a bridge for the system to interact with the user, responsible for receiving user input operands and displaying the final calculation results to the user, while also converting the operand format to meet the requirements of internal calculations. This layer is composed of an input-output layer and a conversion operator module, providing users with a convenient user experience through simple and intuitive interaction.

[0108] S42, convert the operand A and B to the first-order tensor form of the base operand b.

[0109] The conversion algorithm is based on the Qin Jiushao algorithm or iterative division, which represents large integers as: where: represents the input integer, represents the base operand (a system parameter, which can be a power of 2, such as ), is the tensor coefficient, satisfying , is the tensor length, .

[0110] The converted tensor is denoted as , and B is processed in the same way to obtain .

[0111] Using the optimized large number division algorithm, the computing overhead can be reduced.

[0112] S43, serialize the tensor data and send it to the parallel computing layer to ensure data integrity.

[0113] The parallel computing layer is the computing execution unit of the system, mainly responsible for receiving the computing tasks allocated by the scheduling management layer, and performing corresponding computing operations according to different computing stages of the system, including writing the computing results to the specified area of the distributed storage layer, or searching for the required results from the distributed storage layer and performing combined computing, and finally feeding back the computing results to the scheduling management layer or the client interface layer, which is the key link to realize large integer parallel computing.

[0114] Protocol Buffers or Avro can be used for serialization to compress data volume. Through zero-copy network transmission (such as RDMA) to send to the computing node to reduce delay.

[0115] The whole process of steps S3 and S4 includes data generation and data calculation stage.

[0116] Data generation: according to the base operand b, such as b=16, generate each record of the task: the operation result of each element of [1,2,3,...,15] and each element of [1,2,3,...,15];

[0117] Calculation process: convert operands A, B to first-order tensor of base operand b, search the generated data according to the elements of the tensor, and complete the operation of operands A, B.

[0118] Step S5, distribute the search task to the computing node, retrieve the stored data in parallel, verify the search condition, read the effective data from the storage layer, and combine the search results into partial sums.

[0119] In this embodiment, step S5 specifically includes the following steps:

[0120] S51, distribute the search task to the computing node and retrieve the stored data in parallel.

[0121] The search task is , and the scheduling management layer issues the task according to the dynamic allocation strategy. The task parameters are generated by the converted tensor components of the client.

[0122] S52, verify the search condition and read the effective data from the storage layer.

[0123] The search operator first checks the condition (example condition). If it is not satisfied, return 0 directly. Otherwise, locate the storage file according to the bit decomposition of

[0124] read the directory (high​ bits) and files (low bits) in the byte ( calculated).

[0125] verify the header of the data and and extract the value if they match. Use memory-mapped files (mmap) to accelerate reading and reduce I / O latency.

[0126] S53, combine the search results into partial sums, and prepare for aggregation.

[0127] Combination operators include: base operations and sum operations, etc.

[0128] Base operation: perform a left shift operation on each search result , that is equivalent to a binary left shift.

[0129] Sum operation: sum all to get partial results . Use parallel reduction algorithms (such as OpenMP) to complete the combination locally on the computing node.

[0130] Step S5 quickly retrieves pre-stored data through search operators and combines partial results. Through tensor search and pipeline combination calculation, the real-time calculation overhead is reduced.

[0131] Step S6, aggregate the partial sums of all computing nodes to get the final integer result, convert the internal tensor form result to decimal or hexadecimal format, and output the decimal or hexadecimal format large integer data.

[0132] In this embodiment, step S6 specifically includes the following steps:

[0133] S61, aggregate the partial results of all computing nodes to get the final large integer.

[0134] The aggregation operator performs global summation: , where is the number of computing nodes. Use a distributed summation algorithm (such as the Reduce phase of MapReduce) to reduce network overhead through a tree communication mode (such as MPI_Allreduce).

[0135] S62, convert the internal tensor form result to decimal or hexadecimal format.

[0136] The reverse conversion algorithm is based on the Horner rule or iterative multiplication to restore the tensor form to a large integer: ​

[0137] ,in These are the coefficients after aggregation. Use a large number library (such as BN library) for efficient conversion processing.

[0138] S63, the client interface layer outputs large integer data in decimal or hexadecimal format through a web interface or command line.

[0139] It supports formatting enhancements (such as grouping numbers) and logging to ensure a better user experience.

[0140] Figure 2 This is an exemplary system architecture diagram in which the present invention can be applied. For example... Figure 2 As shown, the present invention can be applied to an exemplary system architecture, which includes four modules: a distributed storage layer, a parallel computing layer, and a client interface layer.

[0141] During the data generation phase, the scheduling management layer first obtains system information by reading the system parameter information module, specifically including weighting coefficients ( The system first obtains the basic operand 'b', the number of compute nodes and their IP information, then acquires the node status through the compute node resource acquisition module, and finally uses the dynamic task allocation system to generate tasks from the data. ( , )according to The data is distributed to the parallel computing layer, which performs the calculations using the computation operators and writes the results to the distributed storage layer in a predefined manner. Meanwhile, the scheduling management layer monitors the stored data in real time through the data verification module.

[0142] During the data computation phase, the client interface layer receives decimal / hexadecimal operands A and B input by the user. First, the conversion operator module converts operands A and B into a first-order tensor of system parameter b. Second, the scheduling management layer obtains the computing node status through the computing node resource acquisition module. Then, the dynamic task allocation system module distributes the search task to the parallel computing layer. The search operator module of the parallel computing layer retrieves the search results from the distributed storage layer according to the search task. Next, the data combination operator combines the search results. Finally, the aggregation operator of the scheduling management layer aggregates all the results from the parallel computing layer. Finally, the computation result is displayed to the user through the client interface layer.

[0143] In practice, the tasks of the scheduling management layer include: collecting computing node resources, performing dynamic task allocation, data verification, fault migration, and aggregation operators.

[0144] The collecting computing node resource refers to monitoring the running state of each computing node in real time, and the specific monitoring indexes include CPU utilization rate, memory utilization rate and network delay (communication delay) of the computing node. Through real-time collection and analysis of these indexes, on the one hand, the node resource information is provided for the dynamic task allocation system, so as to reasonably allocate the computing task; on the other hand, the health condition of the computing node can be evaluated, and if it is found that the CPU utilization rate of a computing node is continuously more than 90% for 5 seconds, the memory utilization rate is continuously more than 85% for 5 seconds, or the communication delay is continuously more than 50 milliseconds for 5 seconds, the computing node is marked as a fault node.

[0145] The dynamic task allocation refers to that based on the information of node CPU utilization rate, memory occupation rate and network throughput obtained by the collecting computing node resource method, the weight of each computing node is calculated through a specific weight function, and the expression of the weight function is: After the weight of the computing node is calculated according to the expression, firstly, the computing nodes are sorted according to the weight from high to low, and then the tasks are allocated to the computing nodes in order according to the task size and the remaining resources of the nodes, so as to ensure that the task load of each computing node does not exceed 80% of the maximum processing capacity.

[0146] The data verification is used to guarantee the accuracy and reliability of the data in the distributed storage layer, and the specific operation process is as follows: the scheduling management layer randomly reads 5% of the generated data in the distributed storage layer every 5 minutes for verification. Firstly, the specified content of the read data is compared with the corresponding directory and file information, and the data storage path and identifier are checked to see if they are correct; secondly, the specified content of the read data is compared with the calculation result when the data is initially generated, and the accuracy of the data content is verified. If any of the above two comparisons fails, the hard disk storing the data is marked as a fault hard disk; if the number of healthy hard disks in the system is less than 1, the fault migration process is triggered.

[0147] The fault migration is mainly used to handle the computing node fault and the storage hard disk fault:

[0148] When the collecting computing node resource method finds a fault node, the Checkpoint-Restart mode is used for computing task migration, that is, the unfinished computing task on the fault node is restarted and executed on other healthy nodes through the previously saved checkpoint data, so as to ensure the continuity of the computing task.

[0149] ​When the data checking method finds a faulty hard disk and the number of healthy hard disks is less than 1, the hard disk fault migration process is started, and the specific operation is to add a new hard disk, and then completely copy the data stored in the healthy hard disk to the new hard disk, so as to restore the storage capacity and data reliability of the system, and ensure the normal operation of subsequent data access and calculation.

[0150] The aggregation operator refers to, in the data calculation stage, when all the computing nodes in the system complete the allocated calculation tasks and return the calculation results , the aggregation operator performs summation operation, that is , and outputs the result to the client interface layer, so that the client interface layer shows the calculation result to the user.

[0151] The distributed storage layer is composed of a distributed file system of mobile hard disk arrays, and its main task is to use the distributed storage architecture to store data on multiple mobile hard disks, and through the management mechanism of the distributed file system, to realize the unified management and access of data. In the data storage process, combined with the 3-copy strategy in the data generation stage, the calculation results generated by the computing nodes are stored in different mobile hard disks in the form of 3 copies, and this multi-copy storage method can effectively improve the reliability of data storage. At the same time, the distributed storage architecture can also realize parallel access of data, and multiple computing nodes can read or write data from different hard disks at the same time, greatly improving the data access efficiency and providing strong support for parallel computing of the system.

[0152] The main tasks of the client interface layer include data input and output and conversion operator. The input and output layer performs data input and output tasks.

[0153] Data input and output refers to realizing the data interaction between the system and the user, and the specific functions include receiving the two decimal / hexadecimal operands A, B input by the user, which are the basic data for subsequent integer operations; after the calculation process is completed, it is responsible for displaying the final operation result of the basic number to the user in an intuitive and clear form, so that the user can conveniently obtain the calculation result.

[0154] The conversion operator refers to converting the operands A, B received through data input and output into b-order tensor form, denoted as , , wherein , , , and ,

[0155] , and sends the conversion result to the parallel computing layer.

[0156] The parallel computing layer refers to the layer that receives tasks assigned by the scheduling management layer according to different stages of the system (data generation stage or data computation stage) and performs corresponding computation or data search operations. The specific process is as follows:

[0157] During the data generation phase, the computing nodes in the parallel computing layer receive computing tasks assigned by the scheduling management layer. ,in , After the computing nodes complete the calculations according to the task requirements, they use a 3-replica strategy to write the calculation results into a designated area of ​​the distributed storage layer in a predefined manner.

[0158] The predefined method refers to using the first operand in the computation task. of high The low-order bit is used as the storage directory. The filename is used as a digit, and each calculation result occupies 100 bytes. Each record is in the form of bytes. ,in , .

[0159] During the data computation phase, the computing nodes receive computing tasks assigned by the scheduling management layer. The computing nodes execute the search operator in parallel to obtain... The specific process is as follows:

[0160] First, determine whether the task parameters meet the conditions. This condition is denoted as C.

[0161] If condition C is not met, then The corresponding result is 0. Otherwise, perform the following operation to retrieve data from the distributed storage layer. high The directory corresponding to the bit, low Read the file corresponding to the first bit. The bytes read are denoted as: If the 1st to The value corresponding to the bit and Consistent, the first +1~2 Position and If they match, then the value corresponding to the remaining bits is... The value;

[0162] After the search is completed, the computing nodes perform combined calculations on the search results using data combination operators. This combined calculation process includes basis operations and summation operations in sequence:

[0163] Basis operations refer to the operations performed by the search operator. The result is then left-shifted, i.e.: ;

[0164] The summation operation refers to summing the base operation results according to j, and the result is recorded as

[0165] .

[0166] Finally, the computing node returns the combined calculation result to the scheduling management layer, so that the scheduling management layer performs subsequent aggregation operations.

[0167] Implementing the present embodiment has the following beneficial effects:

[0168] First, it has strong dynamic resource management and fault tolerance capabilities: by collecting real-time resource indicators, it can accurately identify faulty nodes and trigger task migration, while performing data verification and hard disk failure detection to ensure stable operation of the system in complex environments and avoid overall computing interruption due to individual node failure, greatly improving system reliability and availability;

[0169] Second, it can achieve efficient task allocation and load balancing: the weight value of the computing node is calculated based on real-time resource indicators, and the nodes with strong load capacity are selected in descending order to allocate tasks, fully tapping the computing potential of each node, making the resources be reasonably utilized, effectively shortening the calculation time and improving the overall computing efficiency;

[0170] Third, it can support flexible data processing and parallel computing: the user input operand is converted into a first-order tensor form, serialized and sent to the parallel computing layer, and the data generation task is dynamically allocated to realize parallel computing and distributed storage, which can quickly process large-scale integer computing tasks;

[0171] Fourth, it can provide convenient result output and format conversion: after parallel retrieval and storage of data and combination of partial sums, the final integer result can be flexibly converted into decimal or hexadecimal format output to meet the needs of different users for result format, improving the universality and practicality of the method.

[0172] ​The application is operational with numerous general purpose or special purpose computing system environments or configurations. Examples of well- known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through computer readable instructions, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0174] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0175] Embodiment two

[0176] Further reference Figure 3 to the method described above Figure 1 , the application provides an embodiment of an integer parallel computing device based on distributed storage, which corresponds to the method embodiment shown in Figure 1 , and the device can be applied to various electronic devices.

[0177] AsFigure 3 As shown, the embodiment based on distributed storage integer parallel computing device 70 includes identification module 71, allocation module 72, storage module 73, decomposition module 74, combination module 75 and output module 76. Among them:

[0178] The identification module 71 is used to collect real-time resource indicators, including CPU utilization, memory utilization and network delay indicators of each computing node, based on real-time resource indicators, dynamically identify fault nodes, trigger task migration, data verification and hard disk fault detection;

[0179] The allocation module 72 is used to integrate real-time resource indicators, calculate the weight value of each node, arrange the nodes in descending order of weight value, and preferentially select nodes with strong load capacity to allocate tasks;

[0180] The storage module 73 is used to dynamically allocate data generation tasks to computing nodes, perform parallel computing, and distribute and manage parallel computing results;

[0181] The decomposition module 74 is used to obtain the decimal or hexadecimal operands A and B input by the user, convert the operands A and B into the first-order tensor form of the basic operand b, and send the serialized tensor data to the parallel computing layer;

[0182] The combination module 75 is used to allocate search tasks to computing nodes, search stored data in parallel, verify search conditions, read valid data from the storage layer, and combine search results into partial sums;

[0183] The output module 76 is used to aggregate the partial sums of all computing nodes to obtain the final integer result, convert the internal tensor form result into decimal or hexadecimal format, and output the decimal or hexadecimal format large integer data.

[0184] The implementation of the embodiment has the following advantages:

[0185] First, it has strong dynamic resource management and fault tolerance capability: by collecting real-time resource indicators, it can accurately identify fault nodes and trigger task migration, while performing data verification and hard disk fault detection, ensuring stable operation of the system in complex environments, avoiding interruption of overall calculation due to individual node failure, and greatly improving the reliability and availability of the system;

[0186] Second, efficient task allocation and load balancing can be achieved: the weight value of the node is calculated based on the real-time resource indicators, the nodes are arranged in descending order and preferentially selected to allocate tasks, the computing potential of each node is fully tapped, and the resources are reasonably utilized, effectively shortening the calculation time and improving the overall computing efficiency;

[0187] Third, it supports flexible data processing and parallel computing: it converts user-input operands into first-order tensors, serializes them, and sends them to the parallel computing layer, dynamically allocates data generation tasks, realizes parallel computing and distributed storage, and can quickly process large-scale integer computing tasks.

[0188] Fourth, it provides convenient result output and format conversion: after parallel retrieval of stored data and combination of partial sums to obtain the final integer result, it can be flexibly converted into decimal or hexadecimal format output to meet the needs of different users for result format, thereby improving the versatility and practicality of the method.

[0189] Example 3

[0190] To address the aforementioned technical problems, embodiments of the present invention also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0191] The aforementioned computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that only the computer device 8 with components 81, 82, and 83 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0192] The aforementioned computer devices can be desktop computers, laptops, handheld computers, and cloud servers, among other computing devices. These devices can facilitate human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0193] The memory 81 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or a memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 8. Of course, the memory 81 can also include both the internal storage unit and the external storage device of the computer device 8. In this embodiment, the memory 81 is generally used to store an operating system and various application software installed on the computer device 8, such as computer readable instructions of the integer parallel computing method based on distributed storage, etc. In addition, the memory 81 can also be used to temporarily store various data that have been output or will be output.

[0194] The processor 82 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run computer readable instructions or process data stored in the memory 81, such as running computer readable instructions of the integer parallel computing method based on distributed storage.

[0195] The network interface 83 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0196] The implementation of this embodiment has the following beneficial effects:

[0197] First, it has strong dynamic resource management and fault tolerance capability: by collecting real-time resource indicators, it can accurately identify faulty nodes and trigger task migration, while performing data verification and hard disk fault detection, ensuring stable operation of the system in complex environments, avoiding overall computing interruption due to individual node failure, and greatly improving the reliability and availability of the system;

[0198] Second, efficient task allocation and load balancing can be realized: the node weight value is calculated according to the real-time resource index, the nodes with strong load capacity are selected in descending order and preferentially selected to allocate tasks, the computing potential of each node is fully tapped, the resources are reasonably utilized, the computing time is effectively shortened, and the overall computing efficiency is improved;

[0199] Third, flexible data processing and parallel computing can be supported: the user input operand is converted into a first-order tensor form, serialized and sent to the parallel computing layer, data generation tasks are dynamically allocated, parallel computing and distributed storage are realized, and large-scale integer computing tasks can be quickly processed;

[0200] Fourth, convenient result output and format conversion can be provided: after parallel searching and storing data and combining partial sums, the final integer result is obtained, and the decimal or hexadecimal format output can be flexibly converted to meet the needs of different users for the result format, and the universality and practicality of the method are improved.

[0201] Embodiment Four

[0202] The application also provides another embodiment, that is, a computer readable storage medium storing computer readable instructions, the computer readable instructions can be executed by at least one processor to make the at least one processor execute the steps of the distributed storage based integer parallel computing method as described above.

[0203] The implementation of the embodiment has the following beneficial effects:

[0204] First, it has strong dynamic resource management and fault tolerance capability: by collecting real-time resource indicators, the faulty nodes can be accurately identified and task migration is triggered, data verification and hard disk fault detection are performed at the same time, the system can stably run in complex environment, the overall calculation is not interrupted due to individual node failure, and the reliability and usability of the system are greatly improved;

[0205] Second, efficient task allocation and load balancing can be realized: the node weight value is calculated according to the real-time resource index, the nodes with strong load capacity are selected in descending order and preferentially selected to allocate tasks, the computing potential of each node is fully tapped, the resources are reasonably utilized, the computing time is effectively shortened, and the overall computing efficiency is improved;

[0206] Third, flexible data processing and parallel computing can be supported: the user input operand is converted into a first-order tensor form, serialized and sent to the parallel computing layer, data generation tasks are dynamically allocated, parallel computing and distributed storage are realized, and large-scale integer computing tasks can be quickly processed;

[0207] Fourthly, convenient result output and format conversion can be provided: after the stored data is searched in parallel and the parts are combined, the final integer result is obtained, and then the result can be flexibly converted into decimal or hexadecimal format for output, so as to meet the requirements of different users on the result format and improve the universality and practicability of the method.

[0208] Those skilled in the art can clearly understand from the description of the above embodiments that the above-mentioned embodiment methods can be realized by means of software and a necessary general hardware platform, and of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the various embodiment methods of the present application.

[0209] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be realized in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or equivalently replace some of the technical features. Any equivalent structure made by using the content of the present application specification and drawings, directly or indirectly applied to other related technical fields, is also within the patent protection scope of the present application.

Claims

1. A method for parallel computation of an integer based on distributed storage, characterized by, It comprises the following steps: Collecting real-time resource indicators, including CPU utilization, memory utilization and network delay indicators of each computing node, dynamically identifying faulty nodes based on the real-time resource indicators, triggering task migration, data verification and hard disk failure detection; Integrating the real-time resource indicators, calculating the weight value of each node, arranging the nodes in descending order of weight value, and preferentially selecting nodes with strong load capacity to allocate tasks; Dynamically allocate data generation tasks to computing nodes, perform parallel computing, and distribute the parallel computing results for copy management; Obtain the decimal or hexadecimal operands A and B input by the user, convert the operands A and B into the first-order tensor form of the basic operand b, serialize the tensor data and send it to the parallel computing layer; Distribute search tasks to computing nodes, perform parallel search on stored data, verify search conditions, read valid data from the storage layer, and combine search results into partial sums; Sum up the partial sums of all computing nodes to obtain the final integer result, convert the internal tensor form result to decimal or hexadecimal format, and output the decimal or hexadecimal format large integer data.

2. The parallel computation method of an integer based on distributed storage according to claim 1, characterized in that, The step of collecting real-time resource indicators, including CPU utilization, memory utilization and network delay indicators of each computing node, dynamically identifying faulty nodes based on the real-time resource indicators, triggering task migration, data verification and hard disk failure detection comprises: Collecting the real-time resource indicators, including CPU utilization, memory utilization and network delay indicators of each computing node; Based on the real-time resource indicators, dynamically identify faulty nodes, trigger task migration; Periodically verify the integrity and accuracy of the distributed storage layer data, and perform data verification and hard disk failure detection on the distributed storage layer. 3.The parallel computation method based on distributed storage according to claim 1, wherein, The step of integrating the real-time resource indicators, calculating the weight value of each node, arranging the nodes in descending order of weight value, and preferentially selecting nodes with strong load capacity to allocate tasks comprises: Integrate the real-time resource indicators, calculate the weight value of each node; Arrange the nodes in descending order of weight value; Distribute tasks evenly to high-weight nodes to avoid local overload.

4. The parallel computation method of an integer based on distributed storage according to claim 1, characterized in that, The step of dynamically allocating data generation tasks to computing nodes, performing parallel computing, and distributing the parallel computing results for copy management comprises: Dynamically allocate data generation tasks to computing nodes to achieve parallel pre-computation; The computing node performs parallel pre-computation tasks to generate intermediate results; Store the intermediate results to the distributed file system to support fast retrieval and copy management.

5. The parallel computation method of an integer based on distributed storage according to claim 1, characterized in that, The step of obtaining the decimal or hexadecimal operands A and B input by the user, converting the operands A and B into the first-order tensor form of the basic operand b, and serializing the tensor data and sending it to the parallel computing layer comprises: Obtain the decimal or hexadecimal operands A and B input by the user, and parse the operands A and B into a standardized format; Convert the operands A and B into the first-order tensor form of the basic operand b; Serialize the tensor data and send it to the parallel computing layer to ensure data integrity.

6. The parallel computation method of an integer based on distributed storage according to claim 1, wherein, The steps of distributing the search task to the computing nodes, retrieving the stored data in parallel, verifying the search condition, reading the valid data from the storage layer, and combining the search results into partial sums include: Distributing the search task to the computing nodes and retrieving the stored data in parallel; Verifying the search condition and reading the valid data from the storage layer; Combining the search results into partial sums and preparing for aggregation.

7. The parallel computation method of an integer based on distributed storage according to any one of claims 1 to 6, characterized in that, The steps of aggregating the partial sums of all computing nodes to obtain a final integer result, converting the result in the form of an internal tensor into a decimal or hexadecimal format, and outputting the large integer data in the decimal or hexadecimal format include: Aggregating the partial results of all computing nodes to obtain a final large integer; Converting the result in the form of an internal tensor into a decimal or hexadecimal format; The client interface layer outputs the large integer data in the decimal or hexadecimal format through a web interface or a command line.

8. An integer parallel computing device based on distributed storage, characterized by, The method comprises: An identification module configured to collect real-time resource indicators, the real-time resource indicators including CPU utilization, memory utilization, and network delay indicators of each computing node, dynamically identify a faulty node based on the real-time resource indicators, trigger task migration, and perform data verification and hard disk fault detection; An allocation module configured to calculate a weight value of each node based on the real-time resource indicators, arrange the nodes in descending order of the weight values, and preferentially select a node with strong load capacity to allocate a task; A storage module configured to dynamically allocate a data generation task to a computing node, perform parallel computing, and perform distributed storage and replica management on the parallel computing result; A decomposition module configured to obtain decimal or hexadecimal operands A and B input by a user, convert the operands A and B into a first-order tensor form of a basic operand b, and serialize the tensor data and send the serialized tensor data to the parallel computing layer; A combination module configured to distribute a search task to a computing node, retrieve stored data in parallel, verify a search condition, read valid data from a storage layer, and combine search results into partial sums; An output module configured to aggregate the partial sums of all computing nodes to obtain a final integer result, convert the result in the form of an internal tensor into a decimal or hexadecimal format, and output the large integer data in the decimal or hexadecimal format.

9. A computer device, comprising: The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the parallel integer computing method based on distributed storage according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the parallel integer computing method based on distributed storage according to any one of claims 1 to 7.

Images