A Parallel Deep Convolutional Neural Network Optimization Method Based on Winograd Convolution

Abstract

This invention proposes an optimization method for parallel deep convolutional neural networks based on Winograd convolution, comprising: S1, the model batch training stage, employing the feature filtering strategy FF-CSNMI based on cosine similarity and normalized mutual information, which eliminates redundant feature computation by filtering and then fusing, thus solving the problem of excessive redundant feature computation; S2, the parallel parameter update stage, employing the parallel Winograd convolution strategy MR-PWC, which reduces the computational cost of convolution in big data environments by using parallelized Winograd convolution, thereby improving the performance of convolution operations and solving the problem of insufficient convolution operation performance in big data environments; S3, the parameter combination stage, employing the task migration-based load balancing strategy LB-TM, which reduces the average response time of each node in the parallel system by balancing the load among nodes, improving the efficiency of parallel parameter merging, thus solving the problem of low efficiency in parallel parameter merging. This invention significantly improves both parallel efficiency and classification performance.

Description

Technical Field

[0001] This invention belongs to the field of big data classification, and in particular relates to a parallel deep convolutional neural network optimization method based on Winograd convolution. Background Technology

[0002] DCNN (Deep Convolutional Neural Network) is a type of feedforward neural network that incorporates convolutional computations and has a deep, multi-layered structure. It possesses excellent feature extraction and generalization capabilities and is widely used in various fields such as image processing, object tracking and detection, natural language processing, scene classification, and face recognition. However, with the advent of the big data era, the amount of data generated in various fields is exponentially increasing. The training cost of traditional DCNN models increases exponentially with the amount of data, and the complexity of model training also rises due to the increased task complexity in the big data environment. Therefore, designing DCNN algorithms suitable for big data environments is of great significance.

[0003] In recent years, with the widespread application of distributed computing frameworks for big data in DCNN algorithms, Google's MapReduce distributed computing model has gained popularity among scholars due to its advantages in processing big data, such as high fault tolerance and strong scalability. Currently, numerous parallel DCNN algorithms based on the MapReduce framework have been successfully applied to the field of big data analysis and processing. For example, Basic et al. proposed the DC-CNN (Distributed Computing CNN) training algorithm, which combines MapReduce to train the model on each distributed node using parallel computing, realizing the parallel training process of the CNN model on a distributed computing system, achieving higher training efficiency compared to single-machine training. Based on this, Leung et al. designed a parallel parameter update strategy during batch training, proposing the MR-PCNN (Parallel CNN based on MapReduce) algorithm. This algorithm achieves parallel updating of model parameters by parallelizing multiple batch training processes, improving the efficiency of model parameter updates and thus improving the model's training efficiency. To further improve the training efficiency of the model, Wang et al. used parallel matrix multiplication to replace the traditional convolution operation in the parameter update stage and proposed the MR-MPC (Matrix Parallel Computing method based on MapReduce) algorithm. This algorithm improves the operation performance of convolution by parallelizing matrix multiplication, increases the iteration speed of parameter update, and thus improves the training efficiency of the model. Li et al. introduced the idea of ​​centralized parameter server in the parameter combination stage and proposed the MR-CNN (MapReduce-based CNN) training algorithm. This algorithm uses HDFS (Hadoop Distributed File System) as the parameter server to realize the sharing of model parameter updates between nodes, improves the parallel merging efficiency of model parameters, and thus improves the overall training efficiency of the model. Experimental results show that the above four parallel DCNN algorithms have significantly improved the training efficiency of the model, but there are still the following shortcomings: (1) In the batch training process, regardless of whether parallel or non-parallel methods are used, a large number of redundant features will be generated in the training process. If these redundant features are not properly screened, it will bring the problem of excessive redundant feature calculation to the model training. (2) During the parameter update stage, although parallelized matrix multiplication can accelerate the generation of convolution results to a certain extent, the parallelization operation does not change the computational cost of convolution, and the computation process requires frequent IO operations. In the computationally intensive training process, the performance of convolution operation will still be insufficient.(3) During the parameter combination stage, the combination rate of model parameters will be affected by the deviation of the computing power of each distributed node. Therefore, it is easy to cause the phenomenon of mutual waiting between nodes, which prolongs the average response time of each node in the distributed system, resulting in the problem of low efficiency of parameter parallelization and merging in the combination stage. Summary of the Invention

[0004] This invention aims to at least solve the technical problems existing in the prior art, and in particular, it innovatively proposes a parallel deep convolutional neural network optimization method based on Winograd convolution.

[0005] To achieve the above-mentioned objectives of this invention, this invention provides a parallel deep convolutional neural network optimization method based on Winograd convolution, comprising:

[0006] S1, during the batch training phase of the model, the feature filtering strategy FF-CSNMI based on cosine similarity and normalized mutual information is adopted. By filtering first and then fusing, redundant feature calculations are eliminated, thus solving the problem of excessive redundant feature calculations.

[0007] S2, the parallel parameter update stage, adopts the parallel Winograd convolution strategy MR-PWC. By using parallel Winograd convolution, the computational cost of convolution in big data environments is reduced, thereby improving the performance of convolution operations and solving the problem of insufficient performance of convolution operations in big data environments.

[0008] In the S3 parameter combination stage, the task migration-based load balancing strategy LB-TM is adopted. By balancing the load among nodes, the average response time of each node in the parallel system is reduced, and the efficiency of parallelization and merging of model parameters is improved, thereby solving the problem of low efficiency in parallelization and merging of parameters.

[0009] Furthermore, the feature filtering strategy FF-CSNMI based on cosine similarity and normalized mutual information includes:

[0010] S1-1, Feature partitioning: Using the feature similarity metric coefficient FSM based on cosine similarity, the input feature map of the convolutional layer is divided into two parts: the representation feature set P and the redundant feature set R by comparing the size of the feature similarity metric coefficient FSM.

[0011] S1-2, Feature Filtering: To further filter out redundant features in the feature set P, the feature correlation coefficient FAC based on normalized mutual information is used. Based on the FAC value of each feature map in the feature set P, redundant features are iteratively filtered out from the feature set P and included in the redundant feature set R, thereby constructing the final redundant feature set R.

[0012] S1-3, Feature Fusion: After selecting the final redundant feature set R, the redundant features in the redundant feature set R are first fused using the feature fusion function F(Z) to obtain the fused feature set G. Then, by merging the fused feature set G with the representation feature set P, the input feature map of the next layer is constructed.

[0013] Furthermore, the feature similarity metric coefficient (FSM) includes:

[0014] Given that the vector representations of feature map X and mean feature map E are X and E respectively, the feature similarity metric coefficient (FSM) of feature map X relative to mean feature map E is:

[0015] FSM=SIM(X,E)*||norm(XE)|| (1)

[0016] in,

[0017]

[0018]

[0019] SIM(X,E) represents the cosine similarity between X and E;

[0020] ||·|| denotes the first norm of a vector;

[0021] max(XE) represents the maximum value in the XE matrix;

[0022] min(XE) represents the minimum value in the XE matrix.

[0023] Furthermore, the characteristic correlation coefficient FAC includes:

[0024] Given a feature map X and a mean feature map E, and the feature similarity metric coefficient FSM between feature map X and mean feature map E, then the feature correlation coefficient FAC between feature map X and mean feature map E is:

[0025] FAC = NMI(X,E)*FSM (4)

[0026] in,

[0027]

[0028] H(X) is the information entropy of the feature map X;

[0029] H(E) is the information entropy of the feature map E;

[0030] H(X,E) is the joint entropy of feature maps X and E.

[0031] Furthermore, the feature fusion function F(Z) includes:

[0032] Given feature matrices X and Y to be fused, and a fusion feature matrix Z, the feature fusion function F(Z) is:

[0033]

[0034] Where P = [Z1 Z2 … Z] m ], P represents the eigenvector Z i A set;

[0035] Z1 represents the first fused feature vector;

[0036] Z2 represents the second fused feature vector;

[0037] Z m This represents the m-th fused feature vector;

[0038] Λ represents the diagonal elements, which are the eigenvalues ​​of the fused feature matrix Z;

[0039] λ1 represents the first diagonal element;

[0040] λ m This represents the m-th diagonal element;

[0041] Z i =Z X +Z Y ·cosθ i (7)

[0042]

[0043] Z i Let be the fused feature vector, and be the i-th element in P;

[0044] Z X With Z Y These represent the eigenvectors of the feature matrix X and the feature matrix Y, respectively.

[0045] θ i Represents the eigenvector Z X With Z Y The angle between the cosines;

[0046] ||·|| represents the first norm of a vector.

[0047] Furthermore, the parallel Winograd convolution strategy MR-PWC includes:

[0048] S2-1, Feature Map Segmentation: The Multi-Branch Tree Segmentation Strategy (MTS) is used to segment the input feature maps of the convolutional layer, obtaining multi-branch segmentation trees for each feature map, thus creating conditions for parallel computation of parallel Winograd convolution.

[0049] S2-2, Parallel Winograd Convolution: Combine the MapReduce parallel framework to implement parallel computation of Winograd convolution, obtain the convolution results of each convolutional layer, and then store the obtained convolution results in HDFS for use when updating parameters.

[0050] S2-3, Parameter Update: Read the convolution results in HDFS, first use the backpropagation error propagation formula to obtain the weight changes of the DCNN model on each distributed node, then update the parameters of each DCNN model in parallel, and store the weight changes in HDFS for use when combining parameters.

[0051] Furthermore, the multi-branch tree splitting strategy (MTS) includes:

[0052] Using the original feature map as the root node of the multi-branch splitting tree, the equal-size splitting function ESS(X) is first used to split the feature map into sub-feature maps X. 11 ,X 12 ,X 21 ,X 22 , where X 11 ,X 12 ,X 21 ,X 22 Four matrices are represented, and their indices indicate their positions after segmentation. Then, the equally spaced segmentation function EDS(X) is used to construct sub-feature maps D by equally spaced segmentation of the feature maps. Finally, the sub-feature maps X are... 11 ,X 12 ,X 21 ,X 22 And D is a child node of the next level of the multi-branch split tree, and the sub-feature graph X in the above child node is... 11 ,X 12 ,X 21 ,X 22 Perform the same segmentation process until it can no longer be segmented to obtain the multi-branch segmentation tree of the input feature map.

[0053] Furthermore, the equal-size segmentation function ESS(X) includes:

[0054] Given that the size of the original feature map X is h in ×w in The convolution kernel has a size of f×f and a stride of s. The total number of fill pixels on the left and right sides is p0, and the total number of fill pixels on the top and bottom sides is p1. Let 2k = p0 + h. in =p1+w in h in w represents the length of the original feature map X. inThe width of the original feature map X is represented by h; the size of the sub-feature maps after segmentation is h. sub ×w sub Then the equal-size segmentation function ESS(X) is:

[0055]

[0056] Where X 11 ,X 12 ,X 21 ,X 22 These are four sub-feature maps;

[0057] The dimension is h sub ×w sub real numbers;

[0058] The dimension is 2h sub ×2w sub real numbers;

[0059] The equally spaced segmentation function EDS(X) includes:

[0060] Given that the size of the original feature map X is h in ×w in The convolution kernel has a size of f×f and a stride of s. The total number of fill pixels on the left and right sides is p0, and the total number of fill pixels on the top and bottom sides is p1. Let 2k = p0 + h. in =p1+w in If the sub-feature map D constructed by equally spaced feature maps has a size of 2k×2f, then the equally spaced segmentation function EDS(X) is:

[0061]

[0062] in· T Represents the transpose of a matrix;

[0063] D 11 D 21 The block matrix in the sub-feature graph represents the sub-feature graph D, which together constitute the sub-feature graph D.

[0064] Furthermore, the parallel Winograd convolution in S2-2 includes:

[0065] (1) Feature map segmentation and transformation: First, construct multi-branch segmentation trees T for each input feature map in parallel, then traverse the multi-branch segmentation trees T hierarchically to obtain the sub-feature map sequence S. While traversing hierarchically, process the sub-feature map S. i With convolution kernel W k Perform the corresponding Winograd conversion, S i ∈S represents the i-th sub-feature map, Wk This represents the k-th convolutional kernel, and the transformed result is mapped to key-value pairs. i W k Store in HDFS;

[0066] (2) Parallel convolution computation: The Map() function is called to perform parallel Winograd convolution computation. First, based on the key-value pairs of the sub-feature map sequence... i W k Calculate the convolution result Y of each sub-feature map si,k And map the intermediate results of the convolution into key-value pairs. i ,Y si,k Y is temporarily stored in Combine. si,k Sub-feature map S i Convolution kernel W with the k-th channel k The convolution results are then read; next, the intermediate results of the convolution in Combine are read, and the Reduce() function is called to merge the convolution results Y of each sub-feature map in parallel. si Finally, the convolution result Y si Mapped to key-value pairs i ,Y si Store in HDFS, Y si Sub-feature map S i With convolution kernel W k The convolution result;

[0067] (3) Feature map merging: Read the convolution results of each sub-feature map in HDFS, and merge the convolution results of each sub-feature map in parallel according to the order of the sub-feature map sequence S to obtain the final output feature map Y. Then, store the output feature map Y in HDFS for use when updating parameters.

[0068] Further, S3 includes:

[0069] S3-1, Statistical analysis of node information and construction of information retrieval trees: First, statistical analysis of information of all task nodes is performed and sorted in ascending order according to node capacity and stored in array Nodes; then, in order to retrieve node information more quickly and adjust the load between nodes, information retrieval trees (IRT) are used to store the information of task nodes, and the node information in array Nodes is stored in two information retrieval trees (IRT).

[0070] The information retrieval tree is a balanced binary tree with thread pointers. The information stored in each node of the tree consists of the number of each task node, and the information corresponding to each node is stored in an array Nodes indexed by the node number.

[0071] ​​​​S3-2, Setting migration thresholds and constructing migration queues: After constructing information retrieval trees T1 and T2, the task migration route is constructed according to the set migration thresholds. The specific process is as follows: First, a dynamic load migration threshold DLMT is proposed to dynamically set the task migration thresholds, and the upper limit of load migration into T1 is set to L. α (t), the lower limit of load migration for T2 is set to L. β (t); Next, construct the task migration route, first obtaining the node N with the lowest load rate from tree T1. min And obtain the node N with the highest load rate from tree T2. max Next, construct the corresponding migration route, if node N min With node N max If the loads of all nodes meet the migration threshold, then a migration path can be constructed based on the current migration threshold for node N. max The migration task is transferred to node N. min migration route <N max N min > and store the migration route in the migration queue Q; otherwise, abandon the migration route and repeat the above construction process until there are no nodes in T1 and T2 that meet the threshold.

[0072] The Dynamic Load Migration Threshold (DLMT) includes: Given that the median load in the information retrieval trees T1 and T2 at time t is L... m1 (t) and L m2 (t), the average load of T1 and T2 is respectively and In the dynamic load migration threshold, the upper limit for migration in and the lower limit for migration out are L and L respectively. α (t) and L β (t), then the dynamic load migration threshold is:

[0073]

[0074] (3) Execute migration tasks and combine parameters in parallel: After constructing the migration queue, the corresponding migration tasks can be executed according to the current migration route and migration threshold. After the migration tasks are completed, the Reduce() function is called to combine parameters in parallel. The specific process is as follows: First, the head node is taken out from the migration queue Q. <N from N to > Based on the current migration route and migration threshold, from node N from Unprocessed tasks from the middle are sent to node N. to If node N is during the task migration process from With node N to If one node reaches the migration threshold, the current migration task will end; then, the average load of nodes in time t (T1 and T2) will be calculated. and If it appears If the migration ends, the migration process will terminate; otherwise, the migration tasks in the migration queue will continue to be executed until the migration queue is empty. Finally, the Reduce() function is called to process the key-value pairs output by each Map node. <key=W k value=ΔW k Perform parallel merging and use the merged weights as the final weights after training the DCNN model.

[0075] In summary, due to the adoption of the above technical solution, the present invention has the following advantages:

[0076] (1) In the batch training stage, redundant feature calculations are eliminated by screening and then fusion, which solves the problem of excessive redundant feature calculations.

[0077] (2) During the parameter update stage, the computational cost of convolution in the big data environment is reduced by replacing the traditional convolution operation with parallel Winograd convolution, thereby improving the performance of convolution operation and solving the problem of insufficient performance of convolution operation in the big data environment.

[0078] (3) In the parameter combination stage, the average response time of each node in the parallel system is reduced by balancing the load between each node, thereby improving the efficiency of parallelization and merging of model parameters and solving the problem of low efficiency of parallelization and merging of parameters.

[0079] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0080] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0081] Figure 1 This is a schematic diagram of the parallel Winograd convolution of the present invention.

[0082] Figure 2 This is a schematic diagram of the structure of the IRT tree node of the present invention.

[0083] Figure 3 This is a schematic diagram of the construction result of the IRT tree of the present invention.

[0084] Figure 4 This is a schematic diagram showing the speedup ratio of each algorithm in this invention on four datasets.

[0085] Figure 5 This is a schematic diagram illustrating the parallel efficiency of each algorithm in this invention on four datasets.

[0086] Figure 6 This is a schematic diagram illustrating the accuracy of each algorithm in this invention when trained on four datasets.

[0087] Figure 7 This is a schematic diagram showing the running time of each algorithm of the present invention on four datasets. Detailed Implementation

[0088] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0089] This invention proposes a parallel deep convolutional neural network optimization method based on Winograd convolution, which can be widely applied to various fields such as face recognition, object tracking and detection, natural language processing, scene classification, and face recognition.

[0090] Taking its application in the field of facial recognition technology as an example, the process is as follows: Facial data is collected as a training dataset, and this training dataset is input into a DCNN model for training, thereby obtaining a trained DCNN model. The face to be recognized is then input into the trained DCNN model, ultimately achieving the technical effect of rapid face recognition. The DCNN model is trained using a parallel deep convolutional neural network optimization method based on Winograd convolution, including:

[0091] S1, during the batch training phase of the model, the feature filtering strategy FF-CSNMI based on cosine similarity and normalized mutual information is adopted. By filtering first and then fusing, redundant feature calculations are eliminated, thus solving the problem of excessive redundant feature calculations.

[0092] S2, the parallel parameter update stage, adopts the parallel Winograd convolution strategy MR-PWC. By using parallel Winograd convolution, the computational cost of convolution in big data environments is reduced, thereby improving the performance of convolution operations and solving the problem of insufficient performance of convolution operations in big data environments.

[0093] In the S3 parameter combination stage, the task migration-based load balancing strategy LB-TM is adopted. By balancing the load among nodes, the average response time of each node in the parallel system is reduced, and the efficiency of parallelization and merging of model parameters is improved, thereby solving the problem of low efficiency in parallelization and merging of parameters.

[0094] 1. Model batch training

[0095] Currently, in parallel DCNN algorithms under big data environments, a large number of redundant features are generated when using MapReduce to train DCNN models on distributed nodes in parallel batches. These redundant features are not effectively filtered, resulting in excessive computation of redundant features during batch training. Therefore, this paper proposes a feature filtering strategy based on cosine similarity and normalized mutual information (FF-CSNMI) during the batch training stage of the model. This strategy first uses cosine similarity and normalized mutual information to filter out the redundant feature set in the input feature map, and then uses a feature fusion function to fuse the redundant features in the redundant feature set, thereby eliminating the repeated computation of redundant features during batch training and solving the problem of excessive computation of redundant features. The strategy mainly includes three steps: (1) Feature partitioning: a feature similarity metric (FSM) based on cosine similarity is proposed. By comparing the size of the feature similarity metric (FSM), the input feature map of the convolutional layer is divided into two parts: a representation feature set P and a redundant feature set R. (2) Feature filtering: In order to further filter out redundant features in the representation feature set P, a feature association coefficient (FAC) based on normalized mutual information is proposed. According to the size of the FAC value of each feature map in the representation feature set P, redundant features are iteratively filtered out from the representation feature set P and included in the redundant feature set R, thereby constructing the final redundant feature set R. (3) Feature fusion: After filtering out the final redundant feature set R, a feature fusion function F(Z) is proposed. First, the redundant features in the redundant feature set R are fused using the F(Z) function to obtain the fused feature set G. Then, by merging the fused feature set G with the representation feature set P, the input feature map of the next layer is constructed.

[0096] 1.1 Feature Division

[0097] Before feature selection, the input feature map needs to be roughly divided to obtain a preliminary redundant feature set and a representation feature set. The specific process of this division is as follows: First, the input feature map of the convolutional layer is randomly divided into a sampled feature set S and a candidate feature set C according to a certain ratio α, and the mean feature matrix E of the sampled feature set S is calculated. Next, a feature similarity metric coefficient FSM based on cosine similarity is proposed to select redundant features. The feature similarity metric coefficient FSM of each feature map in the candidate feature set C is calculated and sorted in ascending order according to the size of the feature similarity metric coefficient FSM. Finally, based on the size of the feature similarity metric coefficient FSM, the feature map in the candidate feature set C is divided into two parts, a redundant feature set R and a representation feature set P, according to the ratio α from low to high.

[0098] Theorem 1 (Feature Similarity Coefficient, FSM): Given the vector representations of feature map X and mean feature matrix E, respectively, the feature similarity coefficient (FSM) of feature map X with respect to mean feature matrix E is:

[0099] FSM=SIM(X,E)*||norm(XE)|| (1)

[0100] in,

[0101]

[0102]

[0103] SIM(X,E) represents the cosine similarity between X and E;

[0104] ||·|| denotes the first norm of a vector;

[0105] max(XE) represents the maximum value in the XE matrix.

[0106] Proof: The cosine similarity SIM(X,E) represents the degree of similarity between the feature map X and the mean feature matrix E. A larger SIM(X,E) indicates a greater similarity between X and E, and vice versa. norm(XE) represents the normalized matrix of the difference between the feature map X and the mean feature matrix E. Its first norm reflects the degree of offset between the current feature map X and the mean feature matrix E. A smaller value for ||norm(XE)||1 indicates a smaller offset, and vice versa. When the cosine similarity between the feature map X and the mean feature matrix E is small and the offset is high, it indicates that the feature information of the current feature map X is difficult to replace with the feature information of the mean feature matrix E. In this case, the similarity between the feature map X and the mean feature matrix E is low. When the cosine similarity between the feature map X and the mean feature matrix E is large and the offset is low, it indicates that the similarity between the current feature map X and the mean feature matrix E is high. Therefore, the product of the cosine similarity SIM(X,E) and the first norm of the normalized matrix norm(XE) can reflect the similarity between the feature map X and the mean feature matrix E. Q.E.D.

[0107] 1.2 Feature Filtering

[0108] After obtaining the redundant feature set R and the representation feature set P, since the candidate feature set was only roughly divided using a ratio α during the feature partitioning stage, a large number of invalid redundant features remain in the representation feature set P. Therefore, it is still necessary to further filter the redundant features in the representation feature set. The specific filtering process is as follows: First, a feature correlation coefficient FAC based on normalized mutual information is proposed to filter redundant features. The feature correlation coefficient FAC of each feature map in the current representation feature set P relative to the mean feature matrix E is calculated. Then, the feature maps are sorted in descending order according to the size of the feature correlation coefficient FAC, and the feature maps X in the current representation feature set P are divided into redundant feature sets according to a ratio β from high to low. With representation feature set Two parts; finally, the redundant feature set Include it in the redundant feature set R, and calculate the characterization feature set. The mean feature matrix E is obtained by repeating the above steps until the ratio of the current redundant feature set R to the characterization feature set P reaches the expected value η.

[0109] Theorem 2 (Feature Correlation Coefficient FAC): Given a feature map X and a mean feature map E, and the feature similarity metric coefficient FSM of feature map X with respect to the mean feature matrix E, then the feature correlation coefficient FAC of feature map X with respect to the mean feature matrix E is:

[0110] FAC = NMI(X,E)*FSM (4)

[0111] in,

[0112]

[0113] H(X) is the information entropy of the feature map X;

[0114] H(E) is the information entropy of the feature map E;

[0115] H(X,E) is the joint entropy of feature maps X and E.

[0116] Proof: Given that NMI(X,E) represents the normalized mutual information value between feature map X and the mean feature map E, a larger mutual information value indicates a stronger correlation between the current feature map and the mean feature map, and vice versa. The feature similarity metric coefficient (FSM) corresponding to feature map X reflects the similarity between the current redundant feature map X and the mean feature matrix E; that is, a larger FSM value indicates a higher similarity, and vice versa. Therefore, the product of NMI(X,E) and the mean feature map E can, to some extent, measure the correlation of feature information between the current feature map X and the mean feature map E. Specifically, a larger FSM value indicates a high correlation between the current feature map X and the mean feature map E, and vice versa. Q.E.D.

[0117] 1.3 Feature Fusion

[0118] After selecting the final redundant feature set R and representation feature set P, the redundant feature set R and representation feature set P can be fused to construct the input feature map Y of the next layer. The specific process of this fusion is as follows: First, the eigenvectors Z of each feature matrix in the redundant feature set R and representation feature set P are obtained. R Z P And the eigenvalue λ; then, a feature fusion function F(Z) is proposed to fuse redundant features, and the redundant feature matrix in the redundant feature set R is incorporated into the representation feature set P to construct the fused feature set G; finally, the fused feature set G and the representation feature set P are merged to obtain the final input feature map Y.

[0119] Theorem 3 (Feature Fusion Function F(Z)): Given feature matrices X and Y to be fused, and their fusion feature matrix Z, then the feature fusion function F(Z) is:

[0120]

[0121] Where P = [Z1 Z2 … Z] m As shown in equation (15), P represents the eigenvector Z. i A set;

[0122] Z m This represents the m-th fused feature vector;

[0123] Λ represents the diagonal elements, which are the eigenvalues ​​of the fused feature matrix Z;

[0124] Z i =Z X +Z Y ·cosθ i (7)

[0125]

[0126] Z i Let be the fused feature vector, and be the i-th element in P;

[0127] Z X With Z Y Let X and Y represent the eigenvectors of the feature matrices X and Y, respectively. The feature matrix X is the feature matrix to be fused. Here, the feature map is described using the feature matrix to facilitate the subsequent formulation of the feature fusion formula.

[0128] λ1,λ2,…,λ m Represents the eigenvalues ​​of the fused feature matrix Z;

[0129] θ i Represents the eigenvector Z X With Z Y The angle between the cosines.

[0130] Proof: Given that the fused feature matrix Z is obtained by integrating the feature matrix Y with the feature matrix X, and the eigenvalues ​​and eigenvectors of the feature matrix Z are λ and λ, respectively. i With Z i , λ i ∈X=[λ1,λ2,…,λ m ], Z i =Z X +Z Y ·cosθ i Then for the characteristic matrix Z, we have: whether...

[0131] Z×Z i =Z i ×λ i ,i∈[1,m] (9)

[0132] Z i =Z X +Z Y ·cosθ i Because Z X With Z Y Let θ be the eigenvectors of the feature matrices X and Y, respectively, and let θ be the eigenvector Z. X With eigenvector Z Y The angle between the cosines, therefore Z Y ·cosθ i It can be represented as the eigenvector Z Y In the eigenvector Z X Projection Z on Y / X At this point, for the feature vector Z i have:

[0133]

[0134] Z i This represents the i-th eigenvector;

[0135] Z X Represents the eigenvectors of the feature matrix X;

[0136] Z Y Represents the eigenvectors of the feature matrix Y;

[0137] Z Y / X Represents the eigenvector Z Y In the eigenvector Z X Projection on;

[0138] Z i Substituting into equation (9) above, we get:

[0139]

[0140] Since the eigenvalues ​​of the fused feature matrix Z are equal to the eigenvalues ​​of the feature matrix X, therefore for the eigenvector Z... X have:

[0141] X×Z X =Z X ×λ i ,i∈[1,m] (12)

[0142] Z X ×(Z X ) -1 =E, where E represents the identity matrix; at this point, for the eigenvector Z i have:

[0143]

[0144] From equation (13) above, we can see that the eigenvector Z i It is reversible, so multiply the above equation (11) by a (Z) on the right. i ) -1 We can obtain:

[0145] Z = Z i ×λ i ×(Z i ) -1 , i∈[1,m] (14)

[0146] make:

[0147] P = [Z1 Z2 … Z] m (15)

[0148]

[0149] Combining the m equations in equation (14) above, we can obtain:

[0150]

[0151] Q.E.D.

[0152] 2. Parallel parameter updates

[0153] Currently, in parallel DCNN algorithms for big data environments, updating the parameters of DCNN models on distributed nodes requires forward propagation to obtain the convolution results of each convolutional layer, followed by backward propagation to calculate the weight changes of each DCNN model to achieve parallel parameter updates. However, during parameter updates, the computational cost of convolution is enormous, and traditional convolution is inefficient, leading to insufficient convolutional operation performance during parallel parameter updates. Therefore, this paper proposes a parallel Winograd convolution strategy based on MapReduce, using parallel Winograd convolution to replace traditional convolution to improve the model's convolutional operation performance. The strategy mainly includes three steps: (1) Feature map segmentation: a multi-tree segmentation strategy (MTS) is proposed to segment the input feature map of the convolutional layer, and multi-tree segmentation of each feature map is obtained, which creates conditions for parallel computation of parallel Winograd convolution; (2) Parallel Winograd convolution: the parallel computation of Winograd convolution is implemented by combining the MapReduce parallel framework. After obtaining the convolution results of each convolutional layer, the obtained convolution results are stored in HDFS for use when updating parameters; (3) Parameter update: the convolution results in HDFS are read, the error propagation formula of backpropagation is used to calculate the change of weights of the DCNN model on each distributed node, the parameters of each DCNN model are updated in parallel, and the change of weights is stored in HDFS for use when combining parameters.

[0154] 2.1 Feature Map Segmentation

[0155] To achieve parallel computation of Winograd convolution, the input feature maps of the convolutional layers need to be segmented first to meet the requirements of parallel computation. Therefore, the MTS strategy is proposed to segment the input feature maps in parallel, obtaining multi-branch segmentation trees for each input feature map. The specific process is as follows: using the original feature map as the root node of the multi-branch segmentation tree, an equal-size segmentation function ESS(X) is first proposed. The equal-size segmentation of the feature map constructs sub-feature maps, X. 11 ,X 12 ,X 21 ,X 22 Where X 11 ,X 12 ,X 21 ,X 22 Four matrices are represented, and their indices indicate their positions after segmentation. As shown in equation (18); then, the equally spaced segmentation function EDS(X) is proposed, and the equally spaced feature maps are used to construct sub-feature maps D; finally, the sub-feature maps X are... 11 ,X 12 ,X 21 ,X 22And D is a child node of the next level of the multi-branch split tree, and the sub-feature graph X in the above child node is... 11 ,X 12 ,X 21 ,X 22 Perform the same segmentation process until it can no longer be segmented to obtain the multi-branch segmentation tree of the input feature map.

[0156] Theorem 4 (Equal-size segmentation function ESS(X)): Assume the size of the original feature map X is h. in ×w in The convolution kernel has a size of f×f, a stride of s, a total number of fill pixels on the left and right sides of p0, and a total number of fill pixels on the top and bottom sides of p1. Let 2k = p0 + h. in =p1+w in h in w represents the length of the original feature map X. in The width of the original feature map X is represented by h; the size of the sub-feature maps after segmentation is h. sub ×w sub Then the equal-size segmentation function ESS(X) is:

[0157]

[0158] After segmentation, four sub-feature maps X 11 ,X 12 ,X 21 ,X 22 The sizes are h respectively sub ×w sub After being combined into a matrix, the dimension becomes 2h. sub ×2w sub ; The dimension is h sub ×w sub real numbers, The dimension is 2h sub ×2w sub The real number.

[0159] in,

[0160]

[0161]

[0162] in This indicates the specific element of the sub-feature map in the original feature map, and its subscript is used to indicate the position of the element in the original feature map.

[0163] Proof: Given that the size of the input feature map X is h in ×w inThe convolution kernel W has a size of f×f and a stride of s. The input feature map, after being padded, has a size of 2k×2k. Therefore, for the input feature map... The size of the output feature map Y obtained after convolution is h. out ×w out ,in:

[0164]

[0165] X*W=Y (22)

[0166] Where W is the convolution kernel, the input feature map X is the output feature map Y;

[0167] Let the size of the sub-feature map after segmentation be h. sub ×w sub ,in:

[0168]

[0169] Then, for the sub-feature map X obtained after segmentation using the equal-size segmentation function ESS(X) 11 ,X 12 ,X 21 , The output feature maps Y obtained after each convolution operation 11 ,Y 12 ,Y 21 ,Y 22 The size is in:

[0170]

[0171] X 11 *W=Y 11 ,X 12 *W=Y 12 ,X 21 *W=Y 21 ,X 22 *W=Y 22 (25)

[0172] Replacing equation (25) in matrix form yields:

[0173]

[0174] As shown in the sub-feature map X above 11 ,X 12 ,X 21 ,X 22 At the given position in the matrix, the input feature map X is segmented accordingly, ensuring that the size of each sub-feature map is h. sub ×w sub Therefore, we can conclude that:

[0175]

[0176]

[0177] Q.E.D.

[0178] Theorem 5 (Equal Spacing Segmentation Function EDS(X)): Assume the size of the original feature map X is h. in ×w in The convolution kernel has a size of f×f, a stride of s, a total number of fill pixels on the left and right sides of p0, and a total number of fill pixels on the top and bottom sides of p1. Let 2k = p0 + h. in =p1+w in If the sub-feature map D constructed by equally spaced feature maps has a size of 2k×2f, then the equally spaced segmentation function EDS(X) is:

[0179]

[0180] in· T Represents the transpose of a matrix;

[0181] D 11 D 21 This represents the block matrix in the subgraph;

[0182] in,

[0183]

[0184] Proof: Given that the size of the input feature map X is h in ×w in The convolution kernel W has a size of f×f and a stride of s. The input feature map, after being padded, has a size of 2k×2k. Therefore, for the original feature map... The output feature map obtained after convolution operation is in:

[0185]

[0186] X*W=Y (32)

[0187] The sub-feature map X obtained after segmentation using the equal-size segmentation function ESS(X) 11 ,X 12 ,X 21 , The output feature map obtained after convolution operation is This indicates the size of the output feature map, while These represent the dimensions of a single sub-feature map after the convolution operation. Where:

[0188]

[0189]

[0190] Compare the output feature map Y with the output feature map Y X The size indicates that the current output feature map Y X The size of the input feature map Y differs from the size of the output feature map Y by only 1. To ensure the consistency of the output feature map after convolution, the input feature map needs to be further segmented using the midpoint as the starting point to obtain the sub-feature map D. 11 , in:

[0191]

[0192] Sub-feature map D 11 , The output feature map obtained after convolution operation is Y D[1] , in:

[0193]

[0194]

[0195] Output feature map Y D[1] , With output feature map Merge, during the merge, the first column Y D[1] as row vectors Second column Y D[2] As a column vector The resulting output feature map after merging is: in:

[0196]

[0197] Comparing the output feature map Y with the output feature map Y, it can be seen that the output feature map Y C The result is consistent with the result of the output feature map Y, and the output feature map Y C The size is The size h of the output feature map Y obtained after convolving the original feature map X with the original feature map X is... out ×w out Same, that is:

[0198]

[0199] Q.E.D.

[0200] 2.2 Parallel Winograd Convolution

[0201] After using the MTS strategy to segment the input feature map to obtain the multi-branch segmentation tree of each feature map, it can be combined with MapReduce to realize the parallel convolution calculation of Winograd convolution. The calculation process of this parallel Winograd convolution mainly includes three steps: feature map segmentation and transformation, parallel convolution calculation, and feature map merging.

[0202] (1) Feature map segmentation and transformation: First, construct multi-branch segmentation trees T for each input feature map in parallel, then traverse the multi-branch segmentation trees T hierarchically to obtain the sub-feature map sequence S. While traversing hierarchically, process the sub-feature map S. i With convolution kernel W k Perform the corresponding Winograd conversion, S i ∈S represents the i-th sub-feature map, W k This represents the k-th convolutional kernel, and the transformed result is mapped to key-value pairs. i W k Store in HDFS;

[0203] (2) Parallel convolution computation: The Map() function is called to perform parallel Winograd convolution computation. First, based on the key-value pairs of the sub-feature map sequence... i W k Calculate the convolution result Y of each sub-feature map si,k And map the intermediate results of the convolution into key-value pairs. i ,Y si,k Y is temporarily stored in Combine. si,k Sub-feature map S i Convolution kernel W with the k-th channel k The convolution results are then read; next, the intermediate results of the convolution in Combine are read, and the Reduce() function is called to merge the convolution results Y of each sub-feature map in parallel. si Finally, the convolution result Y si Mapped to key-value pairs i ,Y si Store in HDFS, Y si Sub-feature map S i With convolution kernel W k The convolution result of Y is aggregated through the Reduce() function. si,k get.

[0204] (3) Feature map merging: Read the convolution results of each sub-feature map in HDFS, and merge the convolution results of each sub-feature map in parallel according to the order of the sub-feature map sequence S to obtain the final output feature map Y. Then, store the output feature map Y in HDFS for use when updating parameters.

[0205] ​​​​To better illustrate the operation of parallel Winograd convolution, we take a convolutional model with an input feature map size of 7×7, a kernel size of 3×3, a stride of 1, and zero padding pixels on both the left and right sides, as well as zero padding pixels on both the top and bottom sides, as an example. The schematic diagram of performing parallel Winograd convolution on this input feature map is shown below. Figure 1 As shown:

[0206] 2.3 Parameter Update

[0207] After the parallel Winograd convolution calculation is completed, the error propagation term and weight change of each convolutional layer can be calculated in parallel using the backpropagation error propagation formula to achieve parallel parameter update. The specific process of parameter update is as follows: First, the convolutional kernel of the (l-1)th layer is obtained according to the backpropagation formula. Error propagation term Map the results to key-value pairs Store the data in HDFS; then call the Reduce() function to merge the error propagation terms from the backpropagation of each model in HDFS. The error propagation term δ of the (l-1)th layer is obtained. l-1 Finally, calculate the convolution kernel W in the network model of each node. k The change in weight ΔW k And based on the change in weight ΔW k Update convolution kernel W k After taking the parameters in the table, the update result is then mapped to... <W k ,ΔW k The output is sent to HDFS for use in the parameter combination stage.

[0208] 3. Parallel combination of parameters

[0209] Currently, in parallel DCNN algorithms operating in big data environments, merging the parameters of DCNN models on distributed nodes requires waiting for the training of the models on each node to terminate before parameter merging can proceed. However, due to the varying computational capabilities and tasks of each distributed node, mutual waiting is highly likely during parameter combination, leading to low efficiency in parallel parameter merging during the parameter combination phase of the parallel DCNN algorithm. Therefore, this paper proposes a load balancing strategy based on task migration (LB-TM) to improve the efficiency of parallel parameter merging by balancing the task load among nodes. The specific workflow of the LB-TM strategy is as follows:

[0210] (1) Collect node information and construct information retrieval tree. First, collect information on all task nodes and sort them in ascending order according to node capacity and store them in array Nodes. Then, in order to retrieve node information more quickly and adjust the load between nodes, an information retrieval tree (IRT) is designed to store the information of task nodes, and the node information in array Nodes is stored in two information retrieval trees (IRT) (T1 and T2).

[0211] Definition 1 (Information Retrieval Tree, IRT): An information retrieval tree is a balanced binary tree with threaded pointers. The information stored in each node consists of the node numbers of the task nodes, and the information corresponding to each node is stored in an array `Nodes` indexed by its node number. The structure of an IRT tree node is as follows: Figure 2 As shown:

[0212] In this binary tree, the left and right pointers are left_element and right_element, respectively. left_flag and right_flag serve as flags indicating the node types that the left and right pointers point to, and they have the following three states: when the flag is 0, it indicates that it points to the left or right child node; when the flag is 1, it indicates that it points to the predecessor or successor thread node; when the flag is 2, it indicates that it points to the node where the predecessor and successor threads are connected.

[0213] The steps for constructing information retrieval trees T1 and T2 are as follows: ① First, obtain the median load L0(t) in the array Nodes at time t, and use it as the dividing point of node load; ② Then, according to the dividing point L0(t), store the nodes in the array Nodes whose load is lower than L0(t) into T1. In the process of storing nodes, nodes with values ​​higher than L0(t) are stored in T2; ③ In order to adjust the load between nodes more quickly when setting the migration threshold, the total number of nodes in T1 must always be less than the total number of nodes in T2 during the storage process. If this condition is not met when storing nodes, the excess nodes in tree T1 must be moved into T2, and the total load of nodes in T1 must be lower than the total load of nodes in T2; ④ After the storage process is completed, inorder threading can be performed on the information retrieval trees T1 and T2. In order to quickly retrieve node information when constructing the migration queue, the threads between the information retrieval trees T1 and T2 need to be connected, that is, the first predecessor thread of T1 is connected to the last successor thread of T2, and the first predecessor thread of T2 is connected to the last successor thread of T1.

[0214] To more clearly describe the construction process of the information retrieval tree, information retrieval trees T1 and T2 are constructed for a node queue Nodes with a total of 14 nodes. The schematic diagram of the construction result is shown below. Figure 3 As shown:

[0215] (2) Set migration thresholds and construct migration queues. After constructing information retrieval trees T1 and T2, the task migration route can be constructed according to the set migration thresholds. The specific process is as follows: First, a dynamic load migration threshold (DLMT) is proposed to dynamically set the task migration thresholds, and the upper limit of load migration into T1 is set to L. α (t), the lower limit of load migration for T2 is set to L. β (t); Next, construct the task migration route, first obtaining the node N with the lowest load rate from tree T1. min And obtain the node N with the highest load rate from tree T2. max Next, construct the corresponding migration route, if node N min With node N max If the loads of all nodes meet the migration threshold, then a migration path can be constructed based on the current migration threshold for node N. max The migration task is transferred to node N. min migration route <N max N min > and store the migration route in the migration queue Q; otherwise, abandon the migration route and repeat the above construction process until there are no nodes in T1 and T2 that meet the threshold.

[0216] Theorem 6 (Dynamic Load Migration Threshold DLMT) Assume that the median load in the information retrieval tree T1 and T2 at time t is L... m1 (t) and L m2 (t), the average load of T1 and T2 is respectively and In the dynamic load migration threshold, the upper limit for migration in and the lower limit for migration out are L and L respectively. α (t) and L β (t), then the dynamic load migration threshold is:

[0217]

[0218] Proof: Given that information retrieval trees T1 and T2 are both balanced binary trees, and the difference in the number of nodes between them does not exceed 1, therefore, the closer the median load of the two trees is to their respective mean load, the closer the node load in the two trees will be to the overall load mean of the cluster. Right now:

[0219]

[0220] when When the overall average load of the cluster is lower than the median load, the cluster is in a state of unbalanced load. Therefore, the migration limit for T1 is adjusted to L. m1 (t), the lower limit of migration for T2 is L m2 (t), and after migrating tasks between T1 and T2, the average load of the two parts of the T1 and T2 clusters can be made closer to the median load, thus enabling the cluster to achieve load balancing. When At this point, the cluster as a whole tends to be load-balanced, but there are still some nodes with unbalanced loads. In this case, the migration limit of T1 is adjusted to the average value. With median L m1 The maximum value between (t) is used to adjust the lower bound of T2 to the mean. With median L m2 The minimum value between (t) is used to migrate tasks, which brings the average load of clusters T1 and T2 closer together, achieving load balancing. In summary, the Dynamic Load Migration Threshold (DMLT) can control the direction of task migration on each node in the cluster, ensuring overall load balancing. Q.E.D.

[0221] (3) Execute migration tasks and combine parameters in parallel. After constructing the migration queue, the corresponding migration tasks can be executed according to the current migration route and migration threshold. After the migration tasks are completed, the Reduce() function is called to combine parameters in parallel. The specific process is as follows: First, the head node is taken out from the migration queue Q. <N from N to > Based on the current migration route and migration threshold, from node N from Unprocessed tasks from the middle are sent to node N. to If node N is during the task migration process from With node N to If one node reaches the migration threshold, the current migration task will end; then, the average load of nodes in time t (T1 and T2) will be calculated. and If it appears If the migration ends, the migration process will terminate; otherwise, the migration tasks in the migration queue will continue to be executed until the migration queue is empty. Finally, the Reduce() function is called to process the key-value pairs output by each Map node. <key=W k value=ΔW k Perform parallel merging and use the merged weights as the final weights after training the DCNN model.

[0222] 4. Effectiveness of the parallel deep convolutional neural network optimization algorithm based on Winograd convolution (WP-DCNN)

[0223] To verify the performance of the WP-DCNN algorithm, we applied it to four datasets: CIFAR-10, CIFAR-100, SVHM, and Emnist-Balanced. Specific details are shown in Table 1. We compared the WP-DCNN, DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms in terms of parallel performance, parallel efficiency, and accuracy.

[0224] Table 1 Experimental Dataset

[0225] CIFAR-10 CIFAR-100 SVHM Emnist-Balanced Records 60,000 60,000 73,200 131,600 Size 32x32 32x32 32x32 28x28 Classes 10 10 10 47

[0226] 4.1 Parallel Performance Analysis of the WP-DCNN Algorithm

[0227] To evaluate the parallel performance of the WP-DCNN algorithm in a big data environment, this paper conducted five tests on the DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms on the four datasets mentioned above, and used the average of the five runs to calculate the speedup ratio of each algorithm under different numbers of computing nodes. Experimental results are as follows: Figure 4 As shown:

[0228] from Figure 4 It can be seen that when processing the CIFAR-10, CIFAR-100, SVHM, and Emnist-Balanced datasets, the speedup of each algorithm on these four datasets gradually increases with the number of nodes. Furthermore, as the data size gradually expands, the speedup of the WP-DCNN algorithm significantly outperforms the other four algorithms on these datasets. In particular, when processing the relatively small CIFAR-10 dataset, such as… Figure 4As shown in (a), the speedup ratios of the algorithms are not significantly different. When the number of nodes is 2, the speedup ratio of the WP-DCNN algorithm is 1.705, which is 0.081 and 0.168 lower than the speedup ratios of the MR-PCNN and MR-MPC algorithms, respectively, and 0.173 and 0.115 higher than the speedup ratios of the DC-CNN and MR-CNN algorithms, respectively. However, when the number of nodes increases to 8, the speedup ratio of the WP-DCNN algorithm surpasses that of the other four algorithms, which is 0.655, 0.149, 0.328, and 0.520 higher than the speedup ratios of the DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms, respectively. This is because when the number of nodes is small, the communication time between nodes accounts for a large proportion of the overall algorithm runtime, and the improvement in computational efficiency through parallel computing is limited and insufficient to compensate for the communication time consumed by each node during parallel computing. Therefore, the speedup achieved by the WP-DCNN algorithm is relatively small. However, when the number of nodes increases, the WP-DCNN algorithm effectively adjusts the load between nodes through the LB-TM strategy, improving the computational efficiency of each computing node. This results in a speedup ratio for the WP-DCNN algorithm compared to the other four algorithms, hence its slightly higher speedup ratio. On large datasets such as SVHN and Emnist-Balanced, such as... Figure 4 As shown in (c) and 4(d), as the number of nodes increases, the speedup ratios of the other four comparison algorithms gradually stabilize, while the speedup ratio of the WP-DCNN algorithm gradually increases. Especially when processing the Emnist-Balanced dataset, the speedup ratio of the WP-DCNN algorithm generally increases almost linearly, and it always has the highest speedup ratio compared to the other four algorithms. The main reasons for this result are: (1) The WP-DCNN algorithm adopts the LB-TM strategy, which balances the load among nodes, improves the computational performance of the WP-DCNN algorithm, and to a certain extent makes up for the overhead of communication time between nodes; (2) The WP-DCNN algorithm improves the operation performance of convolution during batch training by means of the MR-PWC strategy, which reduces the overall running time of the algorithm. Moreover, as the data scale increases, the advantage of the WP-DCNN algorithm in reducing the overall running time of the algorithm through efficient parallelization of Winograd convolution operations is also gradually amplified. Therefore, the WP-DCNN algorithm has better parallel performance than other algorithms.

[0229] 4.2 Parallel Efficiency Analysis of the WP-DCNN Algorithm

[0230] To evaluate the scalability of the WP-DCNN algorithm in a big data environment, this paper tested DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms on the four datasets mentioned above, and compared the parallel efficiency of each algorithm with different numbers of computation nodes. To avoid the randomness of the test results, each test was run independently five times, and the average of the five parallel efficiencies was taken as the final experimental result. The experimental results are as follows: Figure 5 As shown:

[0231] from Figure 5 It can be seen that the parallel efficiency of all algorithms decreases with the increase of the number of nodes. However, the WP-DCNN algorithm shows the most gradual decrease in parallel efficiency on the large dataset Emnist-Balanced, and consistently maintains the highest parallel efficiency. Specifically, on the smaller datasets CIFAR-10 and CIFAR-100, such as... Figure 5 As shown in (a) and 5(b), the differences in parallel efficiency among the algorithms are relatively small. Furthermore, when the number of nodes is 2, the parallel efficiency of the WP-DCNN algorithm is even lower than that of the MR-MPC algorithm. However, when the number of nodes reaches 6, the parallel efficiency of the WP-DCNN algorithm on the CIFAR-100 dataset surpasses that of other algorithms, exceeding the parallel efficiency of DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms by 0.191, 0.155, 0.029, and 0.075 respectively. This is because when the number of nodes is small, the MR-PWC strategy of the WP-DCNN algorithm provides limited improvement in convolutional operation performance. Moreover, during the feature map merging stage, nodes may wait for each other due to insufficient nodes, further exacerbating the transmission time overhead. However, as the number of nodes increases, this phenomenon is alleviated by the LB-TM strategy, resulting in a significantly shorter overall runtime for the WP-DCNN algorithm compared to other algorithms. Consequently, the parallel efficiency of WP-DCNN only begins to surpass that of other algorithms after the number of nodes increases. When dealing with large datasets like Emnist-Balanced, such as Figure 5As shown in (d), the parallel efficiency of the WP-DCNN algorithm is significantly higher than that of other algorithms, and its parallel efficiency decreases the least. When the number of nodes is 2, 4, 6, and 8, the parallel efficiency of the WP-DCNN algorithm is 0.933, 0.875, 0.812, and 0.745, respectively. The main reasons for this result are: (1) The WP-DCNN algorithm uses the FF-CSNMI strategy in the batch training stage of the model to eliminate the redundant calculation of redundant features in the batch training process, thereby improving the running efficiency of the algorithm; (2) The WP-DCNN algorithm balances the task load among the nodes with the LB-TM strategy. When the number of nodes increases, the WP-DCNN algorithm can better control the load balance of the cluster compared with other algorithms, which results in the most gradual decline in its parallel efficiency and can always maintain the highest parallel efficiency. This shows that the WP-DCNN algorithm has the highest utilization rate of cluster resources and has strong scalability.

[0232] 4.3 Classification Performance Analysis of the WP-DCNN Algorithm

[0233] To evaluate the classification performance of the WP-DCNN algorithm, this paper uses accuracy as the evaluation metric and compares the WP-DCNN algorithm with DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms on four datasets: CIFAR-10, CIFAR-100, SVHM, and Emnist-Balanced. The experimental results are as follows: Figure 6 As shown:

[0234] from Figure 6 It can be seen that as the number of training rounds increases, the classification accuracy of each algorithm gradually rises and then stabilizes. However, compared to DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms, the final accuracy achieved by the WP-DCNN algorithm on all four datasets is higher than that of the other four algorithms. Moreover, the number of training rounds required for the WP-DCNN algorithm to stabilize its classification accuracy is consistently lower than that of the other four algorithms. Specifically, on the CIFAR-10 dataset, for example... Figure 6 As shown in (a), when the number of training epochs is 25, the classification accuracy of the WP-DCNN algorithm has stabilized, and it is 4.62%, 4.22%, 3.09%, and 2.45% higher than the classification accuracy of DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms with the same number of training epochs, respectively. On the CIFAR-100 dataset, as shown in (a), the classification accuracy of the WP-DCNN algorithm has stabilized, and it is 4.62%, 4.22%, 3.09%, and 2.45% higher than that of DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms with the same number of training epochs, respectively. Figure 6As shown in (b), the WP-DCNN algorithm requires 30 training epochs to achieve a stable classification accuracy, which is 10 fewer than the DC-CNN and MR-PCNN algorithms, and 5 fewer than the MR-MPC and MR-CNN algorithms. On the SVHN and Emnist-Balanced datasets, as shown... Figure 6 As shown in (c) and 6(d), the number of training epochs required for the classification accuracy of the WP-DCNN algorithm to plateau is significantly lower than that of the other four algorithms, and the classification accuracy of the WP-DCNN algorithm consistently remains the highest with the same number of training epochs. The main reason for this result is that the WP-DCNN algorithm employs the FF-CSNMI strategy. This strategy eliminates the repeated computation of redundant features during training by fusing redundant features with representational features from the batch training process, thus improving training efficiency and, to some extent, classification accuracy. Therefore, the WP-DCNN algorithm achieves the highest final classification accuracy, and the number of training epochs required for its classification accuracy to plateau is consistently lower than that of the other four algorithms, demonstrating that the WP-DCNN algorithm has excellent classification performance.

[0235] 4.4 Running Time Analysis of the WP-DCNN Algorithm

[0236] To verify the time complexity of the WP-DCNN algorithm, this paper conducted five tests on each of the four datasets mentioned above, comparing the DC-CNN, MR-PCNN, MR-MPC, and MR-CNN algorithms. The average of the five runs was taken as the final experimental result. The experimental results are as follows: Figure 7 As shown:

[0237] from Figure 7It can be seen that the WP-DCNN algorithm consistently maintains the lowest runtime when processing various datasets, and the proportion of WP-DCNN's runtime compared to the other four algorithms gradually increases with the increasing data size. Specifically, when processing the relatively small dataset CIFAR-10, the proportion of WP-DCNN's runtime is 1.42 times, 1.25 times, 1.31 times, and 1.15 times that of DC-CNN, MR-PCNN, MR-MPC, and MR-CNN, respectively. When processing the large dataset Emnist-Balanced, the proportion of WP-DCNN's runtime is 1.85 times, 1.67 times, 1.74 times, and 1.42 times that of DC-CNN, MR-PCNN, MR-MPC, and MR-CNN, respectively. The reasons for this result are: (1) The WP-DCNN algorithm eliminates redundant feature duplication through the FF-CSNMI strategy, reducing the time overhead of the algorithm in redundant feature computation; (2) The MR-PWC strategy improves the convolution operation performance of the algorithm, reduces the time overhead of the algorithm in convolution computation, and as the data scale continues to expand, the parallelization performance of the MR-PWC strategy in convolution operation is further improved, making the WP-DCNN algorithm have better processing capabilities than other algorithms. Therefore, the WP-DCNN algorithm requires the least running time, and as the data scale increases, its running time ratio with the running time of the other four algorithms gradually increases. In summary, the WP-DCNN algorithm has better performance in processing large datasets than other algorithms.

[0238] In summary, our proposed parallel deep convolutional neural network optimization algorithm based on Winograd convolution (WP-DCNN) significantly improves both parallel efficiency and classification performance.

[0239] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A parallel deep convolutional neural network optimization method based on Winograd convolution, characterized in that, include: Collect facial data as a facial training dataset, and input the facial training dataset into the DCNN model for training, thereby obtaining a trained DCNN model; S1, during the batch training phase of the model, the feature filtering strategy FF-CSNMI based on cosine similarity and normalized mutual information is adopted to eliminate the calculation of redundant features by first filtering and then fusing. S1-1, Feature partitioning: Using feature similarity metrics based on cosine similarity. By comparing feature similarity coefficients The size divides the input feature map of the convolutional layer into a representation feature set. With redundant feature set Two parts; S1-2, Feature Filtering: Employing feature correlation coefficients based on normalized mutual information. According to the feature set Each feature map in the middle The magnitude of the value, from the characteristic feature set Redundant features are filtered out in the intermediate iterations and included in the redundant feature set. In this way, the final redundant feature set is constructed. ; S1-3, Feature Fusion: After filtering out the final redundant feature set Next, the feature fusion function is used. Fusion of redundant feature sets Redundant features in the data are used to obtain a fused feature set. Then, by merging and fusing feature sets With representation feature set This constructs the input feature map for the next layer; The feature similarity metric coefficient include: The feature map is divided into a sampled feature set from the input feature map of the convolutional layer. With candidate feature set And calculate the sampling feature set The mean characteristic matrix E before Known feature map With mean feature map The vector representations are respectively and Then the feature map Relative to the mean feature map E before Feature similarity coefficient for: FSM=SIM(X, E before ) ×||norm (X-E before )||1 (1) in, (2) (3) SIM(X, )express and Cosine similarity; max(X- ) represents X- The maximum value in a vector; min(X- ) represents X- The minimum value in the vector; The feature correlation coefficient include: Calculate the mean eigenvalue matrix of the characteristic set P. ; It is known that there is a feature map With mean feature map and feature map Relative to the mean feature map Feature similarity coefficient Then the feature map Relative to the mean feature map Characteristic correlation coefficient for: FAC=NMI(X,E after ) ×FSM (4) in, (5) For feature map Information entropy; H( ) is a feature map Information entropy; H(X, ) is a feature map and feature map The joint entropy; S2, the parallel parameter update stage, adopts the parallel Winograd convolution strategy MR-PWC, which reduces the computational cost of convolution in big data environments by using parallelized Winograd convolution. S3, the parameter combination stage, adopts the task migration-based load balancing strategy LB-TM to reduce the average response time of each node in the parallel system by balancing the load among the nodes.

2. The parallel deep convolutional neural network optimization method based on Winograd convolution according to claim 1, characterized in that, The feature fusion function include: Known feature matrix to be fused and The fusion feature matrix of the two is Then the feature fusion function for: (6) Where P = [Z1 Z2 … Z] m ], Representing the eigenvector A set; This represents the first fused feature vector; This represents the second fused feature vector; Indicates the first One fused feature vector; Representing the diagonal elements, this is the fused feature matrix. eigenvalues; This represents the first diagonal element; Indicates the first One diagonal element; (7) (8) in The fused feature vector is... The first in One element; and Representing the characteristic matrix respectively With characteristic matrix eigenvectors; Representing the eigenvector and The angle between the cosines.

3. The parallel deep convolutional neural network optimization method based on Winograd convolution according to claim 1, characterized in that, The parallel Winograd convolution strategy MR-PWC includes: S2-1, Feature Map Segmentation: The Multi-Branch Tree Segmentation Strategy (MTS) is used to segment the input feature maps of the convolutional layer, obtaining multi-branch segmentation trees for each feature map, thus creating conditions for parallel computation of parallel Winograd convolution. S2-2, Parallel Winograd Convolution: Combine the MapReduce parallel framework to implement parallel computation of Winograd convolution, obtain the convolution results of each convolutional layer, and then store the obtained convolution results in HDFS for use when updating parameters. S2-3, Parameter Update: Read the convolution results in HDFS, first use the backpropagation error propagation formula to obtain the weight changes of the DCNN model on each distributed node, then update the parameters of each DCNN model in parallel, and store the weight changes in HDFS for use when combining parameters.

4. The parallel deep convolutional neural network optimization method based on Winograd convolution according to claim 3, characterized in that, The multi-way tree splitting strategy (MTS) includes: Using the original feature map as the root node of the multi-branch split tree, an equal-size splitting function is first applied. Sub-feature maps are constructed by dividing the feature maps into equal-sized segments. ,in This represents four matrices, with their indices indicating their positions after the partitioning; then, an equal-interval partitioning function is applied. Sub-feature maps are constructed by equally spaced segmentation of the feature map. Finally, the sub-feature map and As a child node of the next layer of the multi-branch split tree, and for the sub-feature maps of the above child nodes. Perform the same segmentation process until it can no longer be segmented to obtain the multi-branch segmentation tree of the input feature map.

5. The parallel deep convolutional neural network optimization method based on Winograd convolution according to claim 4, characterized in that, The equal-size segmentation function include: Known original feature map The size is The size of the convolution kernel is Step size is The total number of fill pixels on the left and right sides is The total number of fill pixels on the top and bottom sides is ,make , Represents the original feature map The length, Represents the original feature map The width; the size of the sub-feature maps after segmentation is Then the equal-size segmentation function for: (18) in These are four sub-feature maps; The dimension is real numbers; The dimension is real numbers; The equally spaced segmentation function include: Known original feature map The size is The size of the convolution kernel is Step size is The total number of fill pixels on the left and right sides is The total number of fill pixels on the top and bottom sides is ,make Sub-feature maps are constructed by equally spaced segmentation of the feature map. The size is Then the equally spaced division function for: (29) in Represents the transpose of a matrix; , This represents the block matrix in the subgraph.

6. The parallel deep convolutional neural network optimization method based on Winograd convolution according to claim 3, characterized in that, The parallel Winograd convolution in S2-2 includes: (1) Feature map segmentation and transformation: First, construct multi-way segmentation trees for each input feature map in parallel. Then traverse the multi-branch split tree level by level. To obtain sub-feature map sequences While traversing the hierarchy, the sub-feature maps are also processed. With convolution kernel Perform the corresponding Winograd conversion. Indicates the first Individual feature maps, Indicates the first Each convolutional kernel is used to map the transformed result into key-value pairs. Store in HDFS; (2) Parallel convolution calculation: The Map() function is called to perform parallel Winograd convolution calculation. First, based on the key-value pairs of the sub-feature map sequence... Calculate the convolution results of each sub-feature map And map the intermediate results of the convolution into key-value pairs. Temporarily stored in Combine. Sub-feature map With the Channel convolution kernel The convolution results are then read; next, the intermediate results of the convolution in Combine are read, and the Reduce() function is called to merge the convolution results of each sub-feature map in parallel. Finally, the convolution result Mapped to key-value pairs Store in HDFS Sub-feature map With convolution kernel The convolution result; (3) Feature map merging: Read the convolution results of each sub-feature map in HDFS, and merge them according to the sub-feature map sequence. The convolution results of each sub-feature map are merged in parallel in sequence to obtain the final output feature map. Then, the output feature map Stored in HDFS for use when updating parameters.

7. The parallel deep convolutional neural network optimization method based on Winograd convolution according to claim 1, characterized in that, S3 includes: S3-1, Statistical analysis of node information and construction of information retrieval trees: First, statistical analysis of information of all task nodes is performed and sorted in ascending order according to node capacity and stored in array Nodes; then, in order to retrieve node information more quickly and adjust the load between nodes, information retrieval trees (IRT) are used to store the information of task nodes, and the node information in array Nodes is stored in two information retrieval trees (IRT). The information retrieval tree is a balanced binary tree with thread pointers. The information stored in each node of the tree consists of the numbers of each task node, and the information corresponding to each node is stored in an array Nodes indexed by the node number.

8. The parallel deep convolutional neural network optimization method based on Winograd convolution according to claim 7, characterized in that, S3 includes: S3-2, Setting migration thresholds and constructing migration queues: After constructing information retrieval trees T1 and T2, the task migration route is constructed according to the set migration thresholds. The specific process is as follows: First, a dynamic load migration threshold (DLMT) is proposed to dynamically set the task migration thresholds, and the upper limit of load migration into T1 is set to... The load migration lower limit for T2 is set to Next, construct the task migration route, starting by obtaining the node with the lowest load rate from tree T1. And obtain the node with the highest load rate from tree T2. Next, construct the corresponding migration route, if the node With nodes If the load of all nodes meets the migration threshold, then a slave node can be constructed based on the current migration threshold. The relocation task has reached the node. migration route And store the migration route in the migration queue. If the migration route is not met, the migration route is abandoned, and the above construction process is repeated until there are no nodes in T1 and T2 that meet the threshold.

9. The parallel deep convolutional neural network optimization method based on Winograd convolution according to claim 8, characterized in that, S3 includes: S3-3, Execute migration tasks and combine parameters in parallel: After building the migration queue, the corresponding migration tasks can be executed according to the current migration route and migration threshold. After the migration tasks are completed, the Reduce() function is called to combine parameters in parallel. The specific process is as follows: First, from the migration queue... Extract the first node of the queue. Based on the current migration route and migration threshold, from the node Unprocessed tasks from the middle are sent to the node. If nodes during task migration With nodes If one node reaches the migration threshold, the current migration task will end; then, the average load of nodes in time t (T1 and T2) will be calculated. and If it appears If the migration fails, the migration process ends; otherwise, the migration tasks in the migration queue continue to be executed until the migration queue is empty. Finally, the Reduce() function is called to process the key-value pairs output by each Map node. Perform parallel merging, and use the merged weights as the final weights after training the DCNN model.

Images