Inference task processing method, system and equipment, medium and program product

Abstract

The invention provides an inference task processing method, system and device, a medium and a program product. The inference task processing method comprises the steps of obtaining an inference task chain; based on the computing power performance of the reasoning computing power chain, performing segmentation processing on the reasoning task chain to obtain a plurality of reasoning task chain segments; each reasoning task chain segment is distributed to a reasoning computing power chain, so that the reasoning computing power chain completes reasoning calculation on the received reasoning task chain segments, and a reasoning result is obtained; wherein the reasoning computing power chain comprises a plurality of third-party idle computing power nodes; and receiving reasoning results returned by the reasoning computing power chains, and determining a target reasoning result of the reasoning task chain according to the reasoning results. Through the technical scheme provided by the invention, the computing power resource is provided for the execution of the reasoning task by the idle computing power node of the third party.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a reasoning task processing method, system, device, medium and program product. Background Art

[0002] In recent years, with the rapid development of general artificial intelligence such as large models and AIGC (Artificial Intelligence Generative Content), the demand for computing power has also increased dramatically, especially GPU (Graphics Processing Unit) computing power. The computing power requirements of general artificial intelligence such as large models and AIGC are mainly divided into two categories: training and reasoning. Training has very strict requirements on computing power, storage, and network, and is generally achieved through centralized deployment of high-performance GPU computing resources. Compared with training, reasoning has much smaller requirements on computing power, storage, and network, and has the conditions for distributed deployment. At the same time, with the rapid development of general AI, the computing power requirements for reasoning will be higher than training. Therefore, there is an urgent need for a solution to meet the computing power requirements for reasoning. Summary of the invention

[0003] The present invention provides a method, system, device, medium and program product for processing reasoning tasks, so as to solve the defect of high computing power requirement for reasoning in the prior art and achieve the effect of idle third-party computing power nodes providing computing power resources for reasoning.

[0004] The present invention provides a method for processing an inference task, comprising: Acquire a reasoning task chain; wherein the reasoning task chain includes a plurality of reasoning tasks; Based on the computing power performance of the inference computing power chain, the inference task chain is segmented to obtain multiple inference task chain segments; Distribute each of the inference task chain segments to one of the inference computing power chains, so that the inference computing power chain completes the inference calculation of the received inference task chain segments and obtains the inference result; wherein the inference computing power chain includes a plurality of idle computing power nodes of a third party; the computing power nodes of the inference computing power chain perform inference calculation in a pipeline parallel manner, or the computing power nodes of the inference computing power chain perform inference calculation in a pipeline parallel manner and each computing power node performs inference calculation in a tensor parallel manner; Receive the reasoning results returned by each of the reasoning computing power chains, and determine the target reasoning result of the reasoning task chain according to the reasoning results.

[0005] According to a method for processing a reasoning task provided by the present invention, before obtaining the reasoning task chain, the method further includes: Obtain node computing power performance of the computing power nodes of multiple third parties; wherein the node computing power performance includes at least one of CPU, GPU, memory, storage performance, network bandwidth, and latency; Based on the node computing power performance, the reasoning computing power chain is constructed.

[0006] According to a method for processing reasoning tasks provided by the present invention, based on the computing power performance of the reasoning computing power chain, before segmenting the reasoning task chain to obtain multiple reasoning task chain segments, the method further includes: Obtaining an inference model, and splitting the inference model to obtain multiple transformer blocks; Performing privacy processing on the transformer block; Based on the available GPU video memory of each computing power node, the multiple transformer blocks that have undergone privacy processing are respectively allocated to each computing power node on the reasoning computing power chain, so that each reasoning computing power chain forms a complete reasoning model.

[0007] According to the inference task processing method provided by the present invention, after receiving the inference results returned by each of the inference computing power chains and determining the target inference result of the inference task chain according to the inference results, the method further includes: Obtaining evaluation parameters of each of the inference computing power chains; wherein the evaluation parameters include resource utilization, performance indicators, stability and integrity; An evaluation coefficient of each of the inference computing power chains is calculated based on the evaluation parameters, and a priority of each of the inference computing power chains is determined according to the evaluation coefficient.

[0008] According to a method for processing reasoning tasks provided by the present invention, the segmenting and distributing each reasoning task chain to one reasoning computing power chain respectively includes: Insert verification tasks into each of the reasoning task chain segments respectively,

[0009] The reasoning task chain segments are homomorphically encrypted, and the homomorphically encrypted reasoning task chain segments are distributed to the reasoning computing power chain.

[0010] According to a method for processing reasoning tasks provided by the present invention, after each of the reasoning task chains is divided into segments and distributed to one of the reasoning computing power chains, the method further includes: Detecting whether each of the computing power nodes in each of the inference computing power chains is abnormal; If an abnormal computing power node is detected, a replacement computing power node corresponding to the abnormal computing power node is determined, and the abnormal computing power node is replaced by the replacement computing power node.

[0011] The present invention also provides an inference task processing system, comprising: Model management module, used to split, distribute and manage inference models; The computing power chain management module is used to build and maintain the inference computing power chain; A task management module is used to form an inference task chain with the received inference tasks, and based on the computing power performance of the inference computing power chain, segment the inference task chain to obtain multiple inference task chain segments, and distribute each of the inference task chain segments to one of the inference computing power chains, so that the inference computing power chain completes the inference calculation of the received inference task chain segments to obtain the inference results; wherein, the inference computing power chain includes multiple third-party idle computing power nodes; the computing power nodes of the inference computing power chain use a pipeline parallel method to perform inference calculations, or the computing power nodes of the inference computing power chain use a pipeline parallel method and each computing power node uses a tensor parallel method to perform inference calculations.

[0012] The present invention also provides an inference task processing device, comprising: A first acquisition module is configured to acquire a reasoning task chain; wherein the reasoning task chain includes a plurality of reasoning tasks; A segment processing module is configured to segment the reasoning task chain based on the computing power performance of the reasoning computing power chain to obtain multiple reasoning task chain segments; A distribution module is configured to distribute each of the inference task chain segments to one of the inference computing power chains, so that the inference computing power chain completes the inference calculation of the received inference task chain segments and obtains the inference result; wherein the inference computing power chain includes a plurality of idle computing power nodes of a third party; the computing power nodes of the inference computing power chain perform inference calculation in a pipeline parallel manner, or the computing power nodes of the inference computing power chain perform inference calculation in a pipeline parallel manner and each computing power node performs inference calculation in a tensor parallel manner; The first determination module is configured to receive the reasoning results returned by each of the reasoning computing power chains, and determine the target reasoning result of the reasoning task chain according to the reasoning results.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described reasoning task processing methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for processing an inference task as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned reasoning task processing methods.

[0016] The inference task processing method, system, device, medium and program product provided by the present invention perform inference calculations through an inference computing power chain composed of multiple third-party idle computing power nodes, thereby realizing the networking of multiple idle computing power resources. Without the need to provide new computing power resources separately, the huge computing power resources required for the current inference task can be met, and the utilization rate of the computing power resources of the idle computing power nodes can be improved. At the same time, when the inference computing power chain performs inference calculations, a pipeline parallel method, or a pipeline parallel and tensor joint processing method can be adopted to quickly complete the inference calculation of the inference task. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the reasoning task processing method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of segmenting the reasoning task chain provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the inference computing chain provided by the present invention for performing inference computing.

[0021] Figure 4 This is one of the schematic diagrams of the reasoning computing power chain provided by the present invention.

[0022] Figure 5 This is the second schematic diagram of the inference computing power chain provided by the present invention.

[0023] Figure 6 It is a structural diagram of the reasoning task processing system provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the reasoning task processing device provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Combine the following Figure 1-Figure 5 The reasoning task processing method of the present invention is described.

[0028] Figure 1 FIG. 1 is a flowchart of a method for processing an inference task according to an exemplary embodiment. Figure 1 As shown, in an exemplary embodiment, the reasoning task processing method includes steps 110 to 140, which are described in detail as follows.

[0029] Step 110, obtaining a reasoning task chain; wherein the reasoning task chain includes a plurality of reasoning tasks.

[0030] In the embodiment of the present invention, the user accesses through the client and submits the reasoning task through the client, and all the received reasoning tasks are formed into a reasoning task chain in chronological order.

[0031] Step 120: based on the computing power performance of the inference computing power chain, segment the inference task chain to obtain a plurality of inference task chain segments.

[0032] In the embodiment of the present invention, the reasoning task chain usually has many reasoning tasks. Using one reasoning computing power chain to perform reasoning calculations may cause the user to wait too long. Therefore, the reasoning task chain can be processed in segments, but the segments of the reasoning task chain after segmentation cannot be too long or too short. If they are too long, the load of each reasoning computing power chain will be unbalanced and the task party will wait too long. If they are too short, the task scheduling will be frequent and the efficiency of the reasoning computing power chain will be low. Therefore, based on the computing power performance of the reasoning computing power chain, the reasoning task chain is processed in segments. When processing in segments, it is necessary to meet the requirement that the waiting time of the task publisher cannot exceed 5 seconds, and the utilization rate of the reasoning computing power chain is not less than 30%.

[0033] Step 130, distribute each of the inference task chain segments to one of the inference computing power chains respectively, so that the inference computing power chain completes the inference calculation of the received inference task chain segments to obtain the inference results; wherein, the inference computing power chain includes multiple third-party idle computing power nodes; the computing power nodes of the inference computing power chain use a pipeline parallel method to perform inference calculations, or the computing power nodes of the inference computing power chain use a pipeline parallel method and each computing power node uses a tensor parallel method to perform inference calculations.

[0034] In the embodiment of the present invention, at least one computing power chain is pre-built, and the computing power chain includes multiple third-party computing power nodes, such as social computing power, family computing power, small data center computing power, etc. Each computing power chain can have complete reasoning computing capabilities. Figure 2 As described above, the inference task chain with 12 inference tasks is segmented into 3 inference task chain segments, and the obtained inference task chain segments are respectively assigned to an inference computing power chain for inference calculation. When performing inference calculation in an inference computing power chain, a pipeline parallel method is used between multiple computing power nodes, and a tensor parallel method can be used between multiple GPU cards in the same computing power node. The relevant introductions of pipeline parallelism and tensor parallelism are as follows.

[0035] Pipeline parallelism means that the inference model is divided into several blocks between layers, and each block is handed over to a computing node. During the propagation process, each computing node passes the intermediate activation to the next computing node. To improve the efficiency of the pipeline, the inference model is divided into many batches, and multiple batches form a pipeline. Each batch is calculated separately, such as Figure 3 As shown, the first letter in each block on the right represents the computing power node, and the second number represents the batch number, such as "D, 0" represents computing power node D and the 0th batch.

[0036] The core problem that pipeline parallelism needs to solve is how to set the batch size. First, consider the case where each computing node has only one GPU card, and define the GPU computing power of the i-th computing node as , the GPU memory bandwidth of the i-th computing node is , the network bandwidth of the i-th computing power node is , batch size is expressed as , embedding size is expressed as , the inference model parameters are , the network protocol delay is , the number of computing nodes in the inference computing chain is .

[0037] The processing time for each token , including the inference time T(inference) and network delay T(net), that is, .

[0038] The inference bottleneck depends on two factors: GPU computing power and GPU memory bandwidth.

[0039] According to the GPU memory bandwidth, the inference time The calculation formula is as follows.

[0040] .

[0041] According to the GPU computing power, the inference time The calculation formula for is as follows.

[0042] .

[0043] Therefore, the inference time is the larger value of the above two calculation methods, and the expression is as follows.

[0044] .

[0045] Simultaneous network delay The calculation formula for is as follows.

[0046] .

[0047] When the inference rate is required to reach (generally expressed as xx / second), it is necessary to meet the following requirements: .

[0048] Will Substitution processing time The expression of , we can get the following expression.

[0049]

[0050] In the above formula, only is an unknown variable, so the maximum value of batch size can be found .

[0051] At the same time, the following requirements are met when the GPU utilization is the highest.

[0052] ; Right now .

[0053] Therefore, the batch size satisfies , and closest to That’s it. This is only applicable to pipeline parallelism, that is, when there is only one GPU card per node.

[0054] Tensor parallelism refers to the division of the inference model within a layer. When the same computing power node has multiple GPUs, the inference calculations within a layer are divided and processed in parallel on multiple GPUs. For example, the tensor parallel GPU cards of each computing power node have Blocks are created by dividing a tensor along a specific dimension into Blocks, each GPU only holds the entire tensor .like Figure 4 As shown, each computing node may include two GUP cards.

[0055] On the basis of the original pipeline parallelism, after superimposing tensor parallelism, the inference time As shown below.

[0056] .

[0057] Define the number of model layers (i.e., the number of KV Cache layers) as , the bandwidth between GPU cards (such as PCIe, NVlink) is .

[0058] The network delay As shown below.

[0059] .

[0060] Time-consuming processing , we can get the following expression.

[0061]

[0062] Find the maximum batch size , because the following expression is satisfied when the GPU utilization is the highest.

[0063] .

[0064] Similarly, the batch size satisfies , and closest to That is, this is the computing power result when there are multiple GPU cards and pipeline parallelism and tensor parallelism are used at the same time.

[0065] After receiving the inference task chain segment, each inference computing chain determines whether to execute the inference task in a pipeline parallel manner or in a pipeline parallel and tensor parallel manner according to the number of its own GPU cards.

[0066] Step 140: receiving the inference results returned by each of the inference computing power chains, and determining the target inference result of the inference task chain according to the inference results.

[0067] In an embodiment of the present invention, the inference result returned by the inference computing power chain forms a target inference result. The target inference result corresponds to the inference task chain, and includes the results corresponding to each inference task in the inference task chain.

[0068] In an embodiment of the present invention, inference calculations are performed by an inference computing power chain composed of multiple third-party idle computing power nodes, so that multiple idle computing power resources can be connected to the grid. Without the need to provide new computing power resources separately, the huge computing power resources required for the current reasoning task can be met, and the utilization rate of the computing power resources of the idle computing power nodes can be improved. At the same time, when the inference computing power chain performs inference calculations, a pipeline parallel method, or a pipeline parallel and tensor joint processing method can be adopted to quickly complete the inference calculation of the reasoning task.

[0069] The technical solution provided by the embodiment of the present invention can meet the needs of various large models, support deployment in various environments, and does not require modification of the original system or application.

[0070] In an exemplary embodiment of the present invention, before acquiring the reasoning task chain, the method further includes the following steps, which are described in detail as follows.

[0071] Obtain node computing power performance of the computing power nodes of multiple third parties; wherein the node computing power performance includes at least one of CPU, GPU, memory, storage performance, network bandwidth, and latency.

[0072] Based on the node computing power performance, the reasoning computing power chain is constructed.

[0073] In the embodiment of the present invention, a corresponding client is installed in the computing node of the third party, and the user logs in to the client to access the reasoning task processing system. The client provides a performance testing tool, which automatically collects the node computing performance of the computing node, including at least one of the CPU, GPU, memory, storage performance, network bandwidth, latency, etc., and reports it to the reasoning task processing system.

[0074] When the computing power node processes and accesses the reasoning task processing system, the reasoning task processing system initializes the stability and integrity of the third-party computing power, and then scores the stability and integrity based on its performance in the reasoning task. Stability refers to the completion rate of the computing power node for the reasoning task, and integrity refers to the accuracy rate of the computing power node in the verification task.

[0075] Multiple third-party computing nodes form an inference computing chain to run the inference calculation of the entire inference model. Each computing node is a third-party computing resource, such as Figure 5 As shown in the figure, each inference model can build multiple inference computing chains.

[0076] When building an inference computing power chain, you can follow the following construction principles.

[0077] (1) The sum of the available GPU memory of all computing nodes in the entire inference computing chain must be greater than the memory requirement of the inference model to complete the inference task, and a certain margin should be left, such as 10%-20%.

[0078] (2) The network bandwidth of each computing node in the inference computing power chain shall not be less than 100 Mb / s, and the network latency shall not be greater than 20 milliseconds. When multiple computing power nodes have the same performance, the computing power node with the smallest latency to the previous computing power node shall be given priority as the next computing power node.

[0079] (3) Set the node threshold. The number of computing nodes in the constructed inference computing power chain does not exceed the node threshold. The node threshold is set based on experience. For example, the number of computing nodes in the entire inference computing power chain generally does not exceed 8.

[0080] (4) The number of GPU cards in each computing power node in the same inference computing power chain should be as similar as possible, and the difference in GPU computing power of each computing power node should be as small as possible. For example, the variance of GPU computing power of computing power nodes should not exceed 20%. Under ideal conditions, the computing power of each computing power node in the inference computing power chain is consistent, which can achieve maximum efficiency.

[0081] (5) The stability and integrity of the computing power nodes shall not be lower than the corresponding thresholds.

[0082] In an exemplary embodiment of the present invention, before the inference task chain is segmented based on the computing power performance of the inference computing power chain to obtain multiple inference task chain segments, the method further includes the following steps, which are described in detail as follows.

[0083] Obtaining an inference model, and splitting the inference model to obtain multiple transformer blocks; Performing privacy processing on the transformer block; Based on the available GPU video memory of each computing power node, the multiple transformer blocks that have undergone privacy processing are respectively allocated to each computing power node on the reasoning computing power chain, so that each reasoning computing power chain forms a complete reasoning model.

[0084] In the embodiment of the present invention, the inference model is split according to the number of transformer blocks in the inference model. Each computing node can carry several transformer blocks, depending on its available GPU memory. In this way, the transformer blocks of the entire inference model can be allocated to each computing node on the computing chain.

[0085] At the same time, in order to ensure data privacy and make data available but invisible, privacy processing is performed on the inference model, such as homomorphic encryption of the transformer block before sending it to the computing power node.

[0086] In an exemplary embodiment of the present invention, after receiving the reasoning results returned by each of the reasoning computing power chains and determining the target reasoning result of the reasoning task chain according to the reasoning results, the method further includes the following steps, which are described in detail as follows.

[0087] Obtain evaluation parameters of each of the inference computing power chains; wherein the evaluation parameters include resource utilization, performance indicators, stability and integrity.

[0088] An evaluation coefficient of each of the inference computing power chains is calculated based on the evaluation parameters, and a priority of each of the inference computing power chains is determined according to the evaluation coefficient.

[0089] In the embodiment of the present invention, all reasoning computing power chains are evaluated and scored regularly to determine the priority of the reasoning computing power chain. The evaluation parameters of each reasoning computing power chain are obtained, and the evaluation parameters include resource utilization U, performance index P, stability S, and integrity I. The weights of each item are superimposed and are Wu, Wp, Ws, and Wi respectively. Resource utilization U is the current GPU resource utilization of the reasoning computing power chain; performance index P is the number of tokens completed per unit time in the historical reasoning tasks of the reasoning computing power chain; stability S is the completion rate of the reasoning computing power chain for the reasoning task; integrity I is the accuracy of the reasoning computing power chain in the verification task.

[0090] The evaluation coefficient Pr of each inference computing power chain is calculated using the formula Pr=Wu×U+Wp×P+Ws×S+Wi×I.

[0091] After determining the priority of each inference computing power chain, when subsequently determining the inference computing power chain corresponding to the inference task chain segment, the inference computing power chain with a higher priority is used first.

[0092] In an exemplary embodiment of the present invention, the step of distributing each of the reasoning task chain segments to one of the reasoning computing power chains includes the following steps, which are described in detail below.

[0093] Insert verification tasks into each of the reasoning task chain segments respectively,

[0094] The reasoning task chain segments are homomorphically encrypted, and the homomorphically encrypted reasoning task chain segments are distributed to the reasoning computing power chain.

[0095] In the embodiment of the present invention, in order to prevent the computing node of a third party from not performing reasoning calculations and arbitrarily fabricating a result for submission, this embodiment adds a verification mechanism, that is, in the obtained reasoning task chain segment, one verification task is superimposed at every interval of a random length. The verification task has a very small demand for computing power and can be completed independently in a computing node with clear results. At the same time, the reasoning task chain segment is also privacy processed, that is, homomorphic encryption is performed to ensure data privacy, so that the data is available but not visible.

[0096] In an exemplary embodiment of the present invention, after distributing each of the reasoning task chain segments to one of the reasoning computing power chains, the method further includes the following steps, which are described in detail as follows.

[0097] Detect whether each of the computing power nodes in each of the inference computing power chains is abnormal.

[0098] If an abnormal computing power node is detected, a replacement computing power node corresponding to the abnormal computing power node is determined, and the abnormal computing power node is replaced by the replacement computing power node.

[0099] In the embodiment of the present invention, the inference computing power chain uses a pipeline parallel method to perform inference calculations. If one of the computing power nodes in the inference computing power chain is disconnected or fails, the subsequent computing power nodes will not be able to complete the inference calculations, which will cause the entire inference computing power chain to be unavailable. Therefore, when using the inference computing power chain for inference calculations, each computing power node is detected to be abnormal. After detecting the presence of an abnormal computing power node, a replacement computing power node is automatically found, and the replacement computing power node completes the inference calculation that the abnormal computing power node should complete.

[0100] Each computing node will cache the intermediate state data of reasoning until the next computing node receives the data from the previous computing node. Once a computing node is abnormal, the newly replaced computing node can continue reasoning directly from the current position without starting from the beginning.

[0101] In an exemplary embodiment of the present invention, after receiving the inference results returned by each inference computing power chain, the inference task processing system first decrypts the inference results with privacy processing. Then the result of the verification task is judged. If the result of the verification task is correct, the decrypted inference result is formed into the target inference result and sent to the user. If the result of the verification task is incorrect, a new inference computing power chain is reselected to complete the inference calculation of this part, and the integrity of the computing power node that executes the verification task is penalized.

[0102] In an exemplary embodiment of the present invention, after obtaining the target reasoning result, the number of tokens is used as a unified settlement unit. In a large model, a token represents the smallest unit of meaning that the model can understand and generate, and is the basic unit of the model. Depending on the specific tokenization scheme used, a token can represent a word, a part of a word, or even just a character.

[0103] The inference task processing system first settles the inference computing power chains using the number of tokens that have been used for inference calculations. It then divides the profits based on the contribution of each computing power node in the inference computing power chain (such as GPU performance, latency, etc.).

[0104] In an exemplary embodiment of the present invention, see Figure 6 , Figure 6 A reasoning task processing system according to an exemplary embodiment includes: Model management module, used to split, distribute and manage inference models; The computing power chain management module is used to build and maintain the inference computing power chain; A task management module is used to form an inference task chain with the received inference tasks, and based on the computing power performance of the inference computing power chain, segment the inference task chain to obtain multiple inference task chain segments, and distribute each of the inference task chain segments to one of the inference computing power chains, so that the inference computing power chain completes the inference calculation of the received inference task chain segments to obtain the inference results; wherein, the inference computing power chain includes multiple third-party idle computing power nodes; the computing power nodes of the inference computing power chain use a pipeline parallel method to perform inference calculations, or the computing power nodes of the inference computing power chain use a pipeline parallel method and each computing power node uses a tensor parallel method to perform inference calculations.

[0105] In an embodiment of the present invention, the model management module manages the splitting, distribution, and upgrading of the inference model, wherein the splitting and distribution are the aforementioned splitting of the inference model into multiple transformer blocks, and based on the available GPU video memory of each computing power node, the multiple transformer blocks that have been privacy processed are respectively distributed to each computing power node on the inference computing power chain.

[0106] The computing power chain management module builds and maintains the inference computing power chain. When an abnormality is detected in a computing power node in the inference computing power chain, it determines the replacement computing power node. The computing power chain management module is also used to calculate the priority of the inference computing power chain and schedule the execution of inference tasks, including pipeline parallelism and tensor parallelism.

[0107] The task management module is used to form an inference task chain based on the inference tasks submitted by the user, randomly superimpose verification tasks in the inference task chain, and allocate inference computing chains to the inference task chain in segments according to the priority of the inference computing chain, so that the inference computing chain can perform the inference task. The relevant instructions for the inference computing chain to perform the inference task have been explained above and will not be repeated here.

[0108] In the embodiment of the present invention, the reasoning task processing system further includes any one of an access authentication module, an encryption and decryption module, and a verification and settlement module. The specific description of each module is as follows.

[0109] The access authentication module is used to implement the access and authentication of third-party computing nodes and users. HIA is used to evaluate the performance of computing nodes through testing tools, as well as to measure and record the resource utilization U, performance index P, stability S, and integrity I of computing nodes.

[0110] The encryption and decryption module is used to implement privacy processing of inference models and inference tasks, as well as to decrypt the inference results submitted by computing nodes.

[0111] The verification and settlement module is used to verify the results of the verification tasks and synchronize the results to the access authentication module. It is also used to realize the settlement of computing power nodes.

[0112] The following is a description of the inference task processing device provided by the present invention. The inference task processing device described below and the inference task processing method described above can be referred to each other. It should be noted that the device provided in the following embodiment and the method provided in the above embodiment belong to the same concept, and the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, which will not be repeated here.

[0113] In an exemplary embodiment of the present invention, see Figure 7 , Figure 7 A device for processing reasoning tasks is shown according to an exemplary embodiment, and includes the following modules.

[0114] The first acquisition module 710 is configured to acquire a reasoning task chain; wherein the reasoning task chain includes a plurality of reasoning tasks.

[0115] The segment processing module 720 is configured to perform segment processing on the reasoning task chain based on the computing power performance of the reasoning computing power chain to obtain multiple reasoning task chain segments.

[0116] The distribution module 730 is configured to distribute each of the inference task chain segments to one of the inference computing power chains respectively, so that the inference computing power chain completes the inference calculation of the received inference task chain segments to obtain the inference results; wherein, the inference computing power chain includes multiple third-party idle computing power nodes; the computing power nodes of the inference computing power chain use a pipeline parallel method to perform inference calculations, or the computing power nodes of the inference computing power chain use a pipeline parallel method and each computing power node uses a tensor parallel method to perform inference calculations.

[0117] The first determination module 740 is configured to receive the reasoning results returned by each of the reasoning computing power chains, and determine the target reasoning result of the reasoning task chain according to the reasoning results.

[0118] In an exemplary embodiment of the present invention, the inference task processing device further includes: A second acquisition module is configured to acquire node computing power performance of the computing power nodes of multiple third parties; wherein the node computing power performance includes at least one of CPU, GPU, memory, storage performance, network bandwidth, and latency; A construction module is configured to construct the reasoning computing power chain based on the node computing power performance.

[0119] In an exemplary embodiment of the present invention, the inference task processing device further includes: A splitting module is configured to obtain an inference model and split the inference model to obtain a plurality of transformer blocks; A privacy processing module, configured to perform privacy processing on the transformer block; The allocation module is configured to allocate the multiple transformer blocks that have undergone privacy processing to each computing node on the inference computing power chain based on the available GPU video memory of each computing power node, so that each inference computing power chain forms a complete inference model.

[0120] In an exemplary embodiment of the present invention, the inference task processing device further includes: A third acquisition module is configured to acquire evaluation parameters of each of the inference computing power chains; wherein the evaluation parameters include resource utilization, performance indicators, stability and integrity; The second determination module is configured to calculate an evaluation coefficient of each of the reasoning computing power chains based on the evaluation parameters, and determine a priority of each of the reasoning computing power chains according to the evaluation coefficient.

[0121] In an exemplary embodiment of the present invention, the distribution module 730 includes: an inserting submodule, configured to insert verification tasks into each of the reasoning task chain segments,

[0122] The homomorphic encryption submodule is configured to perform homomorphic encryption on the reasoning task chain segments, and distribute the homomorphically encrypted reasoning task chain segments to the reasoning computing power chain.

[0123] In an exemplary embodiment of the present invention, the inference task processing device further includes: A detection module, configured to detect whether each of the computing power nodes in each of the inference computing power chains is abnormal; The third determination module is configured to determine a replacement computing power node corresponding to the abnormal computing power node if an abnormal computing power node is detected, and replace the abnormal computing power node with the replacement computing power node.

[0124] Figure 8An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the reasoning task processing method, which includes: obtaining an reasoning task chain; wherein the reasoning task chain includes multiple reasoning tasks; based on the computing power performance of the reasoning computing power chain, segmenting the reasoning task chain to obtain multiple reasoning task chain segments; distributing each of the reasoning task chain segments to one of the reasoning computing power chains, so that the reasoning computing power chain completes the reasoning calculation of the received reasoning task chain segments to obtain the reasoning results; wherein the reasoning computing power chain includes multiple third-party idle computing power nodes; the computing power nodes of the reasoning computing power chain use a pipeline parallel method to perform reasoning calculations, or the computing power nodes of the reasoning computing power chain use a pipeline parallel method and each computing power node uses a tensor parallel method to perform reasoning calculations; receiving the reasoning results returned by each of the reasoning computing power chains, and determining the target reasoning result of the reasoning task chain based on the reasoning results.

[0125] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0126] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the reasoning task processing method provided by the above methods, the method including: obtaining a reasoning task chain; wherein the reasoning task chain includes multiple reasoning tasks; based on the computing power performance of the reasoning computing power chain, segmenting the reasoning task chain to obtain multiple reasoning task chain segments; distributing each of the reasoning task chain segments to one of the reasoning computing power chains, so that the reasoning computing power chain completes the reasoning calculation of the received reasoning task chain segments to obtain an reasoning result; wherein the reasoning computing power chain includes multiple third-party idle computing power nodes; the computing power nodes of the reasoning computing power chain use a pipeline parallel method to perform reasoning calculations, or the computing power nodes of the reasoning computing power chain use a pipeline parallel method and each computing power node uses a tensor parallel method to perform reasoning calculations; receiving the reasoning results returned by each of the reasoning computing power chains, and determining the target reasoning result of the reasoning task chain based on the reasoning results.

[0127] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the reasoning task processing method provided by the above-mentioned methods, the method comprising: obtaining a reasoning task chain; wherein the reasoning task chain comprises a plurality of reasoning tasks; based on the computing power performance of the reasoning computing power chain, segmenting the reasoning task chain to obtain a plurality of reasoning task chain segments; distributing each of the reasoning task chain segments to one of the reasoning computing power chains respectively, so that the reasoning computing power chain completes the reasoning calculation of the received reasoning task chain segments to obtain the reasoning result; wherein the reasoning computing power chain comprises a plurality of third-party idle computing power nodes; the computing power nodes of the reasoning computing power chain perform reasoning calculations in a pipeline parallel manner, or the computing power nodes of the reasoning computing power chain perform reasoning calculations in a pipeline parallel manner and each computing power node performs reasoning calculations in a tensor parallel manner; receiving the reasoning results returned by each of the reasoning computing power chains, and determining the target reasoning result of the reasoning task chain according to the reasoning results.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing an inference task, characterized in that: include: Acquire a reasoning task chain; wherein the reasoning task chain includes a plurality of reasoning tasks; Based on the computing power performance of the inference computing power chain, the inference task chain is segmented to obtain multiple inference task chain segments; Distribute each of the inference task chain segments to one of the inference computing power chains, so that the inference computing power chain completes the inference calculation of the received inference task chain segments and obtains the inference result; wherein the inference computing power chain includes a plurality of idle computing power nodes of a third party; the computing power nodes of the inference computing power chain perform inference calculation in a pipeline parallel manner, or the computing power nodes of the inference computing power chain perform inference calculation in a pipeline parallel manner and each computing power node performs inference calculation in a tensor parallel manner; Receive the reasoning results returned by each of the reasoning computing power chains, and determine the target reasoning result of the reasoning task chain according to the reasoning results.

2. The method for processing reasoning tasks according to claim 1, characterized in that: Before obtaining the reasoning task chain, the method further includes: Obtain node computing power performance of the computing power nodes of multiple third parties; wherein the node computing power performance includes at least one of CPU, GPU, memory, storage performance, network bandwidth, and latency; Based on the node computing power performance, the reasoning computing power chain is constructed.

3. The reasoning task processing method according to claim 1, characterized in that: Before segmenting the inference task chain based on the computing power performance of the inference computing power chain to obtain a plurality of inference task chain segments, the method further includes: Obtaining an inference model, and splitting the inference model to obtain multiple transformer blocks; Performing privacy processing on the transformer block; Based on the available GPU video memory of each computing power node, the multiple transformer blocks that have undergone privacy processing are respectively allocated to each computing power node on the reasoning computing power chain, so that each reasoning computing power chain forms a complete reasoning model.

4. The method for processing reasoning tasks according to claim 1, characterized in that: After receiving the inference results returned by each of the inference computing power chains and determining the target inference result of the inference task chain according to the inference results, the method further includes: Obtaining evaluation parameters of each of the inference computing power chains; wherein the evaluation parameters include resource utilization, performance indicators, stability and integrity; An evaluation coefficient of each of the inference computing power chains is calculated based on the evaluation parameters, and a priority of each of the inference computing power chains is determined according to the evaluation coefficient.

5. The method for processing reasoning tasks according to claim 1, characterized in that: The segmenting and distributing each of the inference task chains to one of the inference computing power chains includes: Insert verification tasks into each of the reasoning task chain segments respectively, The reasoning task chain segments are homomorphically encrypted, and the homomorphically encrypted reasoning task chain segments are distributed to the reasoning computing power chain.

6. The method for processing reasoning tasks according to any one of claims 1 to 5, characterized in that: After distributing each of the reasoning task chain segments to one of the reasoning computing power chains, the method further includes: Detecting whether each of the computing power nodes in each of the inference computing power chains is abnormal; If an abnormal computing power node is detected, a replacement computing power node corresponding to the abnormal computing power node is determined, and the abnormal computing power node is replaced by the replacement computing power node.

7. A reasoning task processing system, characterized in that: include: Model management module, used to split, distribute and manage inference models; The computing power chain management module is used to build and maintain the inference computing power chain; A task management module is used to form an inference task chain with the received inference tasks, and based on the computing power performance of the inference computing power chain, segment the inference task chain to obtain multiple inference task chain segments, and distribute each of the inference task chain segments to one of the inference computing power chains, so that the inference computing power chain completes the inference calculation of the received inference task chain segments to obtain the inference results; wherein, the inference computing power chain includes multiple third-party idle computing power nodes; the computing power nodes of the inference computing power chain use a pipeline parallel method to perform inference calculations, or the computing power nodes of the inference computing power chain use a pipeline parallel method and each computing power node uses a tensor parallel method to perform inference calculations.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the reasoning task processing method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for processing an inference task as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for processing an inference task as claimed in any one of claims 1 to 6 is implemented.

Images