Code distributed computing-based anti-collusion convolutional neural network reasoning acceleration method

Abstract

The invention relates to the technical field of coding distributed computation, convolutional neural networks and information security, and provides an anti-collusion convolutional neural network reasoning acceleration method based on coding distributed computation, which stably accelerates the reasoning of a convolutional neural network, reduces the storage overhead and improves the collusion privacy attack resistance. The method is oriented to a master-slave distributed computing architecture, based on polynomial codes, input tensors and weight tensors of a convolutional layer and a full connection layer are segmented and coded, and coding sub-tensors containing redundant information are generated; then, the coding sub-tensor is sent to a working node for local storage, and then convolution or full-connection calculation is carried out; after each working node completes calculation, a result is sent back to the main node; and the main node recovers the output tensor by using the first received part of calculation results. Compared with an existing numerical stability scheme, the method has lower storage overhead and higher collusion privacy attack resistance.

Description

Technical Field

[0001] This invention relates to the fields of information security, distributed computing, and convolutional neural network technology, specifically to an anti-collusion convolutional neural network inference acceleration method based on coded distributed computing. Background Technology

[0002] Convolutional Neural Networks (CNNs) have been widely applied in various fields due to their superior capabilities in feature abstraction and extraction. Classic models such as AlexNet, VGG-16, and ResNet50 have performed excellently in tasks such as image classification, object detection, and semantic segmentation. Furthermore, with the rapid development of Internet of Things (IoT) devices and the advancement of edge computing technologies, the demand for deploying CNN inference on edge devices is increasing. In practical applications, the effectiveness of CNNs generally improves with increasing model size. This improvement is because deeper networks have more parameters, enabling them to capture more complex patterns and relationships, thereby extracting higher-level abstract features. However, the increasing complexity and number of parameters of CNNs pose significant challenges to their deployment on edge devices, as their computational requirements often exceed the processing capabilities of these devices. For example, the VGG-16 inference process requires 528 MB of memory, while image processing chips like the Arm Cortex A7 CPU only provide 512 MB of memory.

[0003] To address this issue, researchers have conducted extensive research on distributed CNN inference techniques, which utilize model segmentation and data segmentation methods. However, the inference latency of these methods is often significantly affected by slow devices or nodes at the edge, commonly referred to as "falling nodes." The emergence of these falling nodes can be due to communication latency or device failure, which significantly impacts CNN inference performance and weakens the model's accuracy and robustness. Furthermore, because computation needs to be distributed to untrusted distributed computing environments, data owners do not know whether the internal edge devices or nodes are trustworthy. Sending inputs or weights to worker nodes without any protection can lead to serious data privacy breaches, such as when deploying commercial models or inference based on financial or medical data.

[0004] To address the aforementioned issues, Coded Distributed Computing (CDC) was introduced to enhance the robustness of distributed systems while providing some privacy protection. CDC divides computational tasks into... Sub-tasks, utilizing The idea behind erasure coding (among others) Introducing redundancy, using Each worker node performs the computation, so the master node only needs to wait for the fastest one. Number of working nodes (satisfying) The desired result can be obtained by completing the calculation using a threshold (called the recovery threshold), thus significantly reducing computation time. CDC has been successfully applied to matrix multiplication (MM) and other MM-based algorithms, thanks to the ease with which these operations can be linearly decomposed. Furthermore, regarding privacy protection, CDC encodes the data so that individual worker nodes receive only the encoded data and weights; a single node cannot know the original data and weights, thus providing a degree of privacy protection.

[0005] However, the introduction of the CDC scheme also brings some problems. First, due to the complexity of convolutional computation, it is not possible to simply divide the feature map equally to linearly decompose the computation. Furthermore, the tensor dimension in convolutional computation is very high, and element-wise computation is no longer a simple multiplication. Therefore, the matrix multiplication-based CDC scheme cannot be directly extended to the convolutional operations of CNNs, requiring a separate segmentation scheme design. Second, due to the large propagation depth of deep architectures, and the fact that accumulated errors may be significantly amplified with the increase in network layers, numerical stability becomes a key challenge for the application of CDC in CNNs. In addition, although a single node cannot know the original input data and network weights, multiple worker nodes may cooperate to recover the input tensor or weight tensor (a collusive privacy attack), leading to serious privacy leaks. Finally, existing numerically stable CDC schemes are mostly based on outer product segmentation, which incurs significant storage overhead for resource-constrained edge or IoT devices.

[0006] Therefore, it is necessary to propose a numerically stable CDC scheme with high resistance to collusion privacy attacks and apply it to distributed CNN inference, so that the system can safely, correctly and stably accelerate CNN inference while resisting falling behind and preventing collusion privacy attacks. Summary of the Invention

[0007] This invention proposes an anti-collusion convolutional neural network inference acceleration method based on distributed coded computing. It offloads the computation of convolutional and fully connected layers to multiple worker nodes for parallel execution, thereby improving computational efficiency and overcoming the adverse effects of lagging nodes. Compared to existing numerically stable CDC methods, the proposed method has higher anti-collusion privacy attack capabilities and lower storage overhead, and these two advantages can be traded off with anti-lagging capabilities.

[0008] This invention proposes a method for accelerating inference in anti-collusion convolutional neural networks based on distributed coding computation, comprising the following steps:

[0009] Step 1: The master node specifies the number of worker nodes. Segmentation parameters Number of redundant nodes Number of groups Then, the master node explicitly inputs the tensor. and weight tensor Assign a number to each working node ;

[0010] Step 2: The master node is based on the calculation type and partitioning parameters. According to the segmentation algorithm Divided into Size tensor , Divided into ;

[0011] Step 3: The master node, based on... and segmentation parameters Construct the encoding matrix and Then, using all the segmented subtensors... The quadratic combination yields Encoded input subtensors and encoding weight subtensor :

[0012]

[0013]

[0014] Then the master node will and Send to the first in sequence One working node;

[0015] Step 4: Numbered The working node receives the encoded input sub-tensor and weighted subtensor Then, if it's a convolutional layer, calculate... If it is a fully connected layer, then calculate After the calculation is complete, the intermediate results will be... Return to the master node;

[0016] Step 5: The master node determines the parameters. Construct the inner product decoding matrix The master node will Treat it as a group; if all results within the group are received, perform inner product decoding; then check the number of groups currently being decoded; if it is greater than or equal to... Proceed to step 6; otherwise, continue waiting for the result to decode the product of other groups.

[0017] Step 6: The indices of all working nodes involved in inner product decoding form a set. The master node is based on Constructing the decoding matrix Then the master node according to Perform outer product decoding to obtain the subtensor. Then, by concatenating them, the output tensor can be obtained. .

[0018] The benefits of this invention lie in: by matrixing high-dimensional tensor blocks, it lays the foundation for subsequent distributed parallel implementation and facilitates the introduction of redundant information through encoding computation. Based on this, this invention designs an encoded distributed CNN inference scheme based on non-systematic MDS codes for a master-slave distributed architecture, improving the efficiency of CNN inference, reducing storage overhead, and enhancing resistance to collusion privacy attacks. This method segments the input and weight sub-tensors at the master node and then embeds redundant information through encoding. Then, all encoded sub-tensors are unloaded onto worker nodes for computation, and the worker nodes then transmit the computation results back to the master node for final decoding. The redundant embedding introduced by encoding overcomes the problem of missing nodes and stably improves the computational efficiency of convolutional and fully connected layers, thereby accelerating the entire inference process. Compared with existing numerically stable encoded distributed computation, this invention, through its special encoding design, reduces the number of nodes required for inner product decoding, allows for finer tensor segmentation, and enables the stacking of more sub-tensors during encoding, resulting in lower storage overhead and higher resistance to collusion attacks. Inference time and accuracy on different CNN models and datasets demonstrate that this method can safely, stably, and efficiently accelerate inference in convolutional neural networks. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method proposed in this invention.

[0020] Figure 2 This is a schematic diagram of the input and weight tensor segmentation method of the present invention.

[0021] Figure 3 This is a complete framework diagram of the method proposed in this invention.

[0022] Figure 4 This is a comparison chart of the inference time of the CNN inference method in this embodiment of the invention with that of the traditional single-node scheme and the distributed uncoded scheme when there are no missing nodes.

[0023] Figure 5This is a comparison chart of the inference time of the CNN inference method in this embodiment of the invention when there are missing nodes, compared with the inference time of the traditional single-node scheme and the distributed uncoded scheme. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention. Figure 1 As shown, the process framework of a convolutional neural network inference acceleration method based on encoding computation provided by the present invention includes the following specific implementation steps:

[0025] The computations to be accelerated include convolutional layers and fully connected layers. The computation of a convolutional layer refers to the padded input tensor. and weight tensor Step size is The convolution calculation (*) yields the output tensor. The specific definition of convolution calculation is:

[0026]

[0027] in, , This represents the element at the corresponding dimension position, and so on. The computation of a fully connected layer refers to the input tensor... and weight tensor Perform matrix multiplication to obtain the output tensor. .

[0028] First, the master node determines the number of worker nodes participating in the computation. and segmentation parameters Number of redundant nodes Number of groups Number of fallen nodes Then assign the node number as .

[0029] Next, the master node begins to partition the input and weight tensors. If it is a convolutional layer, the master node will partition the input tensor... In dimensions and The above are divided into and The portion will In dimensions and The above are divided into and First, calculate the height of the feature map after convolution. :

[0030]

[0031] Then calculate the starting coordinates of the segmentation. and :

[0032]

[0033] Then calculate the dimensions after segmentation. , , Corresponding size , and :

[0034]

[0035] Then the input subtensor after partitioning It can be represented as:

[0036]

[0037] for Similarly, the starting coordinates of the segmentation are calculated. and :

[0038]

[0039] Then the input subtensor after partitioning It can be represented as:

[0040]

[0041] If it is a fully connected layer, the master node will input tensors. In dimensions and The above are divided into and The portion will In dimensions and The above are divided into and Divide. Then calculate the fractions after division. dimensional size :

[0042]

[0043] Then the input and weight subtensors can be obtained. and :

[0044]

[0045]

[0046] All inputs and weight subtensors after splitting and Represented as:

[0047]

[0048]

[0049] Figure 2 This demonstrates the process of the master node partitioning the input and weight tensors. After partitioning, the master node uses an independent standard Gaussian distribution. generate random numbers and random numbers ,in Then, construct the encoding matrix according to the following rules. and :

[0050]

[0051] Then according to and as well as and Encode the input and weight subtensor to obtain the encoded input:

[0052]

[0053] After the split is complete, the master node will and Send to the Each worker node receives the data and, based on the computation type (convolution or fully connected), calculates the encoded output sub-tensor.

[0054]

[0055] Then it is sent back to the master node. The master node then processes the data based on the parameters. Construct the inner product decoding matrix :

[0056]

[0057] The master node will Consider it as a group. For any Once the master node has collected all the results for this group, it can represent them in the following form, depending on the type of computation:

[0058]

[0059]

[0060] in To represent a noise tensor, we don't need to know its specific composition; we can simply use its symbol. or Represented as From the above formula, we can obtain:

[0061]

[0062] When any Intra-group product decoding successful, using Represents the decoding result, where This indicates that the first to complete inner product decoding The group number. Then the master node... Construct the outer product decoding matrix:

[0063]

[0064] Then it can be expressed in the following form. :

[0065]

[0066]

[0067] Multiply both sides simultaneously inverse matrix We can obtain:

[0068]

[0069] In the After concatenation, the output tensor can be recovered. This data is then used as input for subsequent neural network computations. For pooling and ReLU layers, the master node performs the computation directly without sending data to the worker nodes. A complete WSCC framework diagram is shown below. Figure 3 As shown.

[0070] The effectiveness of this invention in accelerating CNN inference, resisting lag, reducing storage overhead, and improving resistance to collusion attacks is verified by the following experimental data. The experimental code was written in Python and utilized the mpi4py (3.1.6) library for distributed computation. All tensors were represented using 32-bit floating-point numbers from PyTorch, "torch.float32". Convolutional and fully connected functions were based on a basic, unoptimized implementation in PyTorch (2.4.1-CPU). To simulate lag nodes, an artificial delay was introduced using the "sleep()" function in Python, and a random function was used to specify the index of the lag node. All experiments were conducted on an Alibaba Cloud ecs.c8ae.large instance (2 vCPU, 4GB RAM) with all PyTorch multithreading optimizations disabled. The experiments used the AlexNet model, trained on the NWPU-RESISC45 remote sensing image classification dataset. The distributed unencoded (dis) scheme partitions the input and weight tensors using the method described in the paper, then sends the task to worker nodes for computation, and finally returns the result to the master node, without any encoding or decoding. The native scheme, on the other hand, performs the computation directly on the master node.

[0071] Regarding the inference acceleration effect of CNN, the experimental results are as follows: Figure 4 and Figure 5 As shown. The experiment uses 8 working nodes for computation, and is set... Number of fallen nodes , , When there are no lagging nodes, the inference time of the WSCC scheme is slightly higher than that of the distributed uncoded scheme, but compared with the native scheme, the inference time is still reduced by about 86%, achieving an acceleration of about 7.4 times. The Conv1 layer requires padding due to the non-integer partitioning of the input channels, resulting in a slightly lower acceleration effect; however, the computation time is still reduced by about 79%, achieving an acceleration of nearly 5 times. In addition, the average computation time of all convolutional layers (Conv2-5) and all fully connected layers (Fc1-3) is reduced by about 86%, achieving an acceleration of about 7.4 times.

[0072] For cases where nodes fall behind, 10 worker nodes are used for computation, and a distributed, uncoded scheme is configured. WSCC scheme settings Number of fallen nodes , The experimental results are shown in the figure. Because the distributed uncoded scheme utilizes all available worker nodes for computation, the task is divided into smaller subtasks, resulting in a lower average computation time than the WSCC scheme when no nodes are missing. However, once a missing node appears, the average computation time of the distributed scheme far exceeds that of the WSCC scheme, which is unacceptable. Nevertheless, the WSCC scheme still achieves an 86% reduction in average computation time, providing approximately a 7.4x speedup. Therefore, the WSCC scheme demonstrates robustness in the face of missing nodes.

[0073] The experimental results regarding reduced storage overhead are shown in Table I. This experiment compares the storage overhead of the WSCC scheme with existing numerically stable schemes. All comparison schemes are based on a configuration of one master node and 16 worker nodes. For the WSCC scheme, the partitioning parameters of ConvL 1 are set to... The remaining ConvL partition parameters are set to For the CRME, RKRP, and ORTHOPOLY schemes, the partitioning parameters for convolutional layers 1-5 are set to... For fully connected layers, the partitioning parameter is set to Compared to other numerically stable schemes, the WSCC scheme significantly reduces storage overhead, thanks to the advantages of inner product partitioning. Regarding storage overhead, although the number of input channels in Conv1 limits the partitioning, the WSCC scheme still reduces storage overhead by at least 36% compared to other schemes. For ConvL2-5, storage overhead is reduced by at least 82% compared to other schemes. For FcL1-3, since other schemes only partition the weight tensor, while the WSCC scheme partitions both the weight and input tensors, the storage overhead of the weight tensor remains unchanged, but the storage overhead of the input tensor is reduced by at least 93%.

[0074] Table I:

[0075]

[0076] The experimental results regarding resistance to collusion attacks are shown in Table II. This experiment compared the resistance to collusion attacks of different numerically stable schemes with the same number of working nodes. The scheme with the fewest working nodes indicates that it guarantees the weight tensor and input tensor can resist collusion attacks. - Minimum number of working nodes required for a conspiracy attack. Experimental results show that the WSCC scheme requires fewer working nodes than all other numerically stable schemes. As shown in Table 2, the number of working nodes required by the WSCC, ORTHOMATDOT, and COMPLEX MATDOT schemes varies with... The number of working nodes required by WSCC increases linearly, but it is only 50%-60% of the number required by the other two schemes. In contrast, the number of working nodes required by the CRME, RKRP, ORTHOPOLY, and COMPLEX POLY schemes increases linearly. A second increase. When When the number of nodes is ≥3, the number of nodes in the quadratic growth scheme is significantly higher than that in the linear growth scheme. Therefore, the WSCC scheme demonstrates superior resistance to collusion attacks.

[0077] Table II:

[0078] .

Claims

1. A collusion-resistant convolutional neural network inference acceleration method based on encoding distributed computing, characterized in that, The method comprises the following steps: Step 1: Master node determines number of worker nodes , split parameters , number of redundant nodes , number of groups Then, master node determines input tensor and weight tensor Assigns number to each worker node ; Step 2: Master node splits the tensor based on computation type and split parameters , according to the split algorithm into sub-tensors , into ; Step 3: The master node constructs the encoding matrix and the splitting parameters according to the encoding matrix and ; then it performs linear combinations of all the split sub-tensors to obtain encoding input sub-tensors and encoding weight sub-tensors : ， ， Then the master node will and in turn to the first worker node; Step 4: The worker node numbered receives the encoded input sub-tensor and the weight sub-tensor Then, if it is a convolutional layer, it calculates , and if it is a fully connected layer, it calculates After the calculation is completed, the intermediate result is returned to the master node; Step 5: Master node constructs inner product decoding matrix according to parameters ; Master node considers as a group, and if it receives all the results within the group, it performs inner product decoding;​ The current number of groups being coded is then checked, and if greater than or equal to Step 6 is performed, otherwise the process continues to wait for the results to code the inner product for the other groups. Step 6: the indices of all the worker nodes participating in the inner product decoding form a set , the master node constructs a decoding matrix according to , and then the master node performs outer product decoding according to , and obtains a sub-tensor . After splicing, the output tensor is obtained.

2. The method of claim 1, wherein, The computation of a convolutional layer refers to the input tensor after padding. and weight tensor Step size is The convolution calculation yields the output tensor. ; In step 2, the height of the convolved feature map is first calculated : ， The starting coordinates of the division are then calculated and : ， After calculating the dimensions after the split , , Corresponding dimensions , and : ， So the split input sub-tensors may be represented as: ； For , the split start coordinates and are also calculated: ， So the split input sub-tensors may be represented as: ； If it is a fully connected layer, the master node will split the input tensor into and parts respectively along the dimensions and , and split the input tensor into and parts respectively along the dimensions and , and then calculate the dimension size of the split ， Then the input and weight sub-tensors are obtained and : ， ； all input and weight sub-tensors after splitting and is represented as: ， 。 3. The method of claim 2, wherein, In step 3, the construction process of the encoding matrix comprises: The master node is from an independent standard Gaussian distribution generated a random number and a random number , wherein ; The encoding matrix is then constructed according to the following rules and : 。 4. The method of claim 3, wherein, In step 5, the master node constructs the inner product coding matrix H according to the parameters In the following way : 。 5. The method of claim 4, wherein, In step 5, the process of the inner product decoding comprises: The master node will consider as a group, for any As long as the master node collects all the results of this group, according to the different types of calculation, the master node represents them in the form as follows: ， ， wherein denotes the noise tensor, and or denotes as From the above equation, it follows that ， When any The group number of the group that first completes the inner product decoding is represented by The decoding result is represented by The decoding result is represented by The decoding result is represented by 6. The method of claim 5, wherein, In step 6, the master node constructs the outer product decoding matrix as follows: The outer product decoding matrix is constructed as follows: ， Further, the following is obtained: ， After stitching, the output tensor is recovered and used as input for the subsequent neural network computation; if it is a pooling layer and a relu layer, the master node directly completes the computation without sending to the worker node.

Images