Semi-supervised manifold binary neural network construction method under optimal scale

Abstract

The invention discloses a method for constructing a semi-supervised manifold binary neural network under an optimal scale, which effectively reserves spatial information of data and makes up for information loss by combining an OLSR method with binary convolution, thereby improving the accuracy and efficiency of depth feature extraction. By integrating scale items in the OLSR model, the method can better adapt to data under different scales, so that the generalization ability and adaptability of the model are improved; and a scale self-learning method is adopted to ensure that an optimal solution is obtained in each iteration, so that the convergence speed of the model is accelerated, and the training efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and specifically relates to a method for constructing a semi-supervised manifold binary neural network at an optimal scale. Background Art

[0002] In the field of deep learning, especially in computer vision tasks, model compression and acceleration have always been a focus of attention. Traditional deep neural networks often have a large number of parameters and complex structures, and consume a lot of computing resources when processing large-scale data, which limits their application in embedded systems, mobile devices, and resource-constrained environments. Therefore, researchers have been looking for various methods to compress and accelerate deep neural networks to improve their efficiency and performance in practical applications.

[0003] As an effective model compression method, quantization technology effectively reduces storage and computing costs by converting the weights and activation values ​​of the network from high precision to low precision. Among them, binary neural network (BNN), as an extreme form of quantization technology, limits the weights and activation values ​​of the network to +1 and -1, greatly reducing the complexity of computing and storage operations. However, although BNN has made significant progress in compression models, its extremely simplified representation has led to a significant decline in the expressiveness and performance of the model.

[0004] In order to solve the problem of BNN performance degradation, researchers have proposed various improvement methods. Among them, the optimization-based binary neural network adopts a series of strategies, such as minimizing quantization error, improving network loss function, reducing gradient error, etc., to improve the performance and generalization ability of the model. In terms of training strategy optimization of binary neural networks, Liu et al. explored that Adam can better handle the information loss of binary neural networks through its adaptive learning rate strategy and has better generalization ability. Xu et al. proposed to introduce the ReCU module to update invalid weights during training, and use an adaptive exponential scheduler to identify the range of invalid weights. Experiments have shown that this method has good results. Tu et al. proposed AdaBin to better fit different distributions, improve feature expression capabilities by adaptively obtaining the best binary set of weights and features for each layer, and introduce a gradient-based optimization method to obtain parameters to achieve end-to-end training.

[0005] However, although these methods have made some progress in improving the performance of BNN, there are still some challenges and limitations. For example, the training process of the model may become more complicated and time-consuming, requiring more computing resources and optimization techniques. In addition, how to retain the information and structural characteristics of the data as much as possible while keeping the model compressed is still a problem that needs further research.

[0006] The performance degradation of BNN leads to the loss of basic information and the inability to accurately represent the spatial structural characteristics of the data. In this regard, the OLSR model in manifold learning plays an important role. The OLSR model imposes orthogonality restrictions on the transformation matrix of the least squares regression (LSR), effectively learns the structural information of the data, and improves the classification results. Finally, a closed solution is obtained by iterative minimization of the error function. Therefore, the OLSR model is of great significance in manifold learning. Since the OLSR model has the characteristics of effective preservation of data structure information and fast convergence, it is expected to process the deep features obtained after binary convolution by combining it with the optimal scaling technology.

[0007] Therefore, future research directions may include but are not limited to: exploring more efficient and accurate model compression and acceleration methods, further optimizing the BNN and OLSR models based on optimization, and exploring the application of deep learning models in new application scenarios and fields. These efforts will help promote the widespread application and further development of deep learning technology in practical applications. Summary of the invention

[0008] In order to overcome the shortcomings of the prior art, the present invention provides a method for constructing a semi-supervised manifold binary neural network at an optimal scale. By combining the OLSR method with binary convolution, the spatial information of the data is effectively retained, and information loss is compensated, thereby improving the accuracy and efficiency of deep feature extraction. By integrating the scale term in the OLSR model, it can better adapt to data at different scales, thereby improving the generalization ability and adaptability of the model; the scale self-learning method is adopted to ensure that the optimal solution is obtained in each iteration, thereby accelerating the convergence speed of the model and improving the training efficiency.

[0009] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0010] Step 1: Data preprocessing;

[0011] Split the dataset X into vector labels X l and contains n u X of unknown labels u , that is, X=[X l ,X u ], where n l +n u =n;

[0012] Define the function as follows:

[0013]

[0014] Step 2: Get the final data;

[0015] Q = sign(AX) = sign(A[X l ,X u ])=sign([AX l ,AX u ])=[Q l ,Q u ] (1)

[0016] in A represents the mapping matrix, Q l represents the preprocessing result of labeled data, Q u Represents the preprocessing result of unknown label data;

[0017] Step 3: Define the category indicator matrix;

[0018] Category indicator matrix The details are as follows:

[0019]

[0020] Where C represents the number of classes;

[0021] Step 4: Feature extraction OLSR regression model;

[0022] Let the projection matrix W∈R m*c , W T W = I, and the OLSR model for feature extraction is as follows:

[0023]

[0024] Step 5: Orthogonal least squares regression model with optimal scale;

[0025] Due to the orthogonality constraint, the subspace W in equation (4) has a fixed scale. In order to solve the problem of scale variation, the scale term α is introduced into the equation, so that the orthogonal least squares regression problem with the optimal scale is expressed as:

[0026]

[0027] Step 6: Define the loss of labeled data;

[0028] For the labeled dataset Q l Using cross entropy loss, the loss is defined as:

[0029]

[0030] where Y i represents the true label value, Y i ' is the predicted label value, H(·) represents the cross entropy, and l represents the number of labeled data;

[0031] Step 7: Define the loss of unlabeled data; for unlabeled data, use the same processing steps as labeled data to obtain the predicted label Y of the unlabeled data j '; At the same time, use OLSR to learn the projection transformation matrix and obtain the predicted label Y j "; For Q u , the loss is defined as:

[0032]

[0033] Where u represents the number of unlabeled data;

[0034] Step 8: Optimize the total loss function;

[0035] The total loss function is defined as:

[0036] L=L(Q l )+β(t)L(Q u ) (7)

[0037] Where β(t) is the coefficient that balances the two loss terms; the formula for β(t) is:

[0038]

[0039] Among them, T1 and T2 represent β t The hyperparameters of the changes, β f Represents β t The upper limit value of , t represents the number of model iterations;

[0040] Preferably, in step 5, the specific steps for solving the orthogonal least squares regression problem are as follows:

[0041] Step 5-1: Input: binary data matrix Q∈R m*n , label matrix Y l ; The number of unlabeled data n u , the number of classes C;

[0042] Step 5-2: Initialization: Random orthogonal matrix L T L=LL T =I and α=1;

[0043] Step 5-3: Repeat the following steps until the convergence condition is met:

[0044] (1) W = L(:, 1:C) and H = L(:, (C+1):m);

[0045] (2) Calculate Y u pass

[0046] (3) Calculate γ by

[0047] (4) Update α by

[0048] (5) Calculate B by B = Q[Y, αQ T H]∈R m*n ;

[0049] (6) Update L by In B, singular value decomposition is performed;

[0050] Step 5-4: Output: projection matrix W, unknown label matrix Y u .

[0051] The beneficial effects of the present invention are as follows:

[0052] 1. The proposed SSMBNN-OS runs effectively on datasets. It performs well on COIL20, MNIST, UMIST, and AT&T. It achieves the second highest accuracy after 100 rounds of training. The scale term not only speeds up the convergence speed but also significantly improves the classification accuracy.

[0053] 2. SSMBNN-OS achieves the second highest accuracy on COIL100 and MNIST. Despite 20% less labeled data for COIL100 and 50% less labeled data for MNIST, the accuracy only slightly drops by 0.20% and 0.11%, respectively. This confirms the superiority of SSMBNN-OS and its advantage on less labeled data.

[0054] 3. In COIL100, the loss and accuracy of the original BNN, IR-Net, RAD, and SSMBNN-OS are compared. The results show that SSMBNN-OS can achieve convergence in just 5 iterations, while the others require more iterations. This proves the fast convergence of SSMBNN-OS. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of the method of the present invention.

[0056] Figure 2 It is the overall framework diagram of the method of the present invention.

[0057] Figure 3 It is the accuracy curve of SSMBNN and SSMBNN-OS on COIL20 according to the embodiments of the present invention.

[0058] Figure 4(a) to (h) are curve graphs showing the changes in loss and accuracy of RAD, IR-Net, ReCU, DWR, SiMaN, SD-BNN, original BNN and SSMBNN-OS as the training process progresses on the COIL100 dataset in the embodiments of the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0060] The main purpose of the present invention is to solve the shortcomings of current technology in extracting data structure information and to introduce a new optimal scale semi-supervised binary neural network manifold (SSMBNN-OS) by taking advantage of the effectiveness of the OLSR method. Specifically, this invention aims to:

[0061] Improve the extraction efficiency and accuracy of data structure information: By combining the OLSR method with binary convolution, the spatial information of the data is effectively retained and the information loss is compensated, thereby improving the accuracy and efficiency of deep feature extraction.

[0062] Solve the problem of lack of scale variation in subspace: By integrating scale terms in the OLSR model, it can better adapt to data at different scales, thereby improving the generalization ability and adaptability of the model.

[0063] Improve model training efficiency and convergence speed: Use scale self-learning method to ensure that the optimal solution is obtained in each iteration, thereby accelerating the convergence speed of the model and improving training efficiency.

[0064] Overall, this invention aims to improve the accuracy and efficiency of data structure information extraction, thereby contributing to the progress in the fields of deep learning and data analysis.

[0065] This paper proposes a method for constructing a semi-supervised manifold binary neural network under optimal scale to solve the problem that BNN loses key information during the binarization process, resulting in poor performance. Due to the effective preservation and fast convergence characteristics of OLSR data structure information, we combine it with optimal scaling technology to process the deep features obtained after binary convolution. This method can improve the performance of the network and the robustness of the system. Figure 1 and Figure 2 , the implementation steps of the semi-supervised manifold binary neural network with optimal scale of the present invention are as follows:

[0066] Step 1: Data preprocessing. Use orthogonal constraints to retain more local information and divide the dataset X into vector labels X l and contains n u X of unknown labels u Therefore, X=[X l ,Xu ], where n l +n u =n. The function is defined as follows:

[0067]

[0068] Step 2: Get the final data.

[0069] Q = sign(AX) = sign(A[X l ,X u ])=sign([AX l ,AX u ])=[Q l ,Q u ] (10)

[0070] in

[0071] Step 3: Define the category indicator matrix. Category indicator matrix The details are as follows:

[0072]

[0073] Step 4: Feature extraction OLSR regression model. Since LSR does not consider the structure of local data. Therefore, this paper first proposes an orthogonal constraint as a solution to retain more local information. Assume that the projection matrix W∈R m*c and W T W = I, and the obtained OLSR model for feature extraction is as follows:

[0074]

[0075] Step 5: Orthogonal least squares regression model with optimal scale. Due to the orthogonality constraint, the subspace W in equation (4) has a fixed scale. In order to solve the problem of scale variation, the scale term α is introduced into the equation. This allows the new orthogonal least squares regression problem with optimal scale to be expressed as:

[0076]

[0077] The specific solution steps are as follows:

[0078] 1. Input: binary data matrix Q∈R m*n , label matrix Y l The number of unlabeled data n u , the number of classes c;

[0079] 2. Initialization: Random orthogonal matrix L T L=LL T =I and α=1;

[0080] 3. Repeat steps 4-9 until the convergence condition is met:

[0081] 4. W = L(:, 1: C) and H = L(:, (C+1): m);

[0082] 5. Calculate Y u pass

[0083] 6. Calculate Y by

[0084] 7. Update alpha pass

[0085] 8. Calculate B by B = Q[Y, αQ T H]∈R m*n ;

[0086] 9. Update L by In B, singular value decomposition is performed;

[0087] 10. Output: projection matrix W, unknown label matrix Y u .

[0088] Step 6: Define the loss of labeled data. l Using cross entropy loss, the loss is defined as:

[0089]

[0090] where Y i ' is the predicted value, and H(·) represents the cross entropy.

[0091] Step 7: Define the loss for unlabeled data. For unlabeled data, use the same processing steps as for labeled data to obtain the predicted label Y' for the unlabeled data. j At the same time, OLSR is used to learn the projection transformation matrix and obtain the predicted label Y" j For Q u Generally speaking, the loss can be defined as:

[0092]

[0093] Step 8: Optimize the total loss function. The total loss function is defined as:

[0094] L=L(Q l )+β(t)L(Q u ) (16)

[0095] Where β(t) is a coefficient that balances the two loss terms. If β(t) is initially set too high, the performance of the network will degrade because the predicted label Y'j and Y" j Most of them are incorrect. Therefore, when the network is stable, the value of β(t) should be gradually increased. The SA algorithm is used to reduce the risk of local minima during the optimization process. The formula of β(t) is:

[0096]

[0097] Example:

[0098] 1. Description of simulation conditions and data set;

[0099] The present invention selects five benchmark data sets to test the feasibility of the SSMBNN-OS algorithm. These data sets are:

[0100] The UMIST dataset introduced and cited in the paper “C.Hou, F.Nie, X.Li, D.Yi, and Y.Wu, Joint embedding learning and sparse regression: A framework for unsupervised feature selection, IEEE Transactions on Cybernetics, vol.44, no.6, pp.793–804, 2013.”;

[0101] The COIL20 dataset introduced and used in the paper “SANene, SK Nayar, H. Murase, et al., Columbia object image library (coil-20), 1996.”;

[0102] The COIL20 dataset introduced and used in the document "Columbia Object Image Library (COIL-100)," SANene, SK Nayarand H. Murase, Technical Report CUCS-006-96, February 1996;

[0103] The MNIST dataset introduced and used in the paper “Y.LeCun, The mnist database of handwritten digits, http: / / yann.lecun.com / exdb / mnist / , 1998.”

[0104] The summary information of the above datasets is shown in Table 1.

[0105] Table 1. Summary information of benchmark datasets

[0106]

[0107] 2. Simulation content;

[0108] This simulation uses a clustering method to cluster the learned deep features and depth contrast map representations, and calculates the evaluation index introduced and used in the literature "ZFKnops, JAMaintz, MAViergever, and JPPluim, Normalized mutual information based registration using k-means clustering and shading correction, Medical image analysis, vol.10, no.3, pp.432–439": Accuracy (ACC).

[0109] In order to compare the effectiveness of the present invention, the comparison methods were selected as follows:

[0110] The RAD method mentioned by Ding et al. in the paper "R. Ding, T.-W. Chin, Z. Liu, and D. Marculescu, Regularizing activation distribution for training binarized deep networks, in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, pp. 11 408–11 417."

[0111] IR-Net method mentioned by Qin et al. in the paper “H. Qin, R. Gong, X. Liu, M. Shen, Z. Wei, F. Yu, and J. Song, “Forward and backward information retention for accurate binary neural networks,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 2250–2259.”

[0112] ReCU method mentioned in the paper "Z.Xu, M.Lin, J.Liu, J.Chen, L.Shao, Y.Gao, Y.Tian, ​​and R.Ji, "Recu:Reviving the dead weights in binary neural networks," in Proceedings of the IEEE / CVF international conference on computer vision, 2021, pp.5198–5208." by Xu et al.

[0113] The DWR method proposed by Xue et al. in the literature “P.Xue,Y.Lu,J.Chang,X.Wei,and Z.Wei,“Fast and accurate binary neural networks based on depth-width reshaping,”in Proceedings of theAAAI Conference on Artificial Intelligence,vol.37,no.9,2023,pp.10 684–10692.”;

[0114] SiMaN method mentioned by Lin et al. in the paper “M.Lin, R.Ji, Z.Xu, B.Zhang, F.Chao, C.-W.Lin, and L.Shao, “Siman: Sign-to-magnitude network binarization,” IEEE Transactions on PatternAnalysis & Machine Intelligence, vol.45, no.05, pp.6277–6288, 2023.”;

[0115] And the SD-BNN mentioned by Xue et al. in the document “P. Xue, Y. Lu, J. Chang, X. Wei, and Z. Wei, “Self-distribution binary neural networks,” Applied Intelligence, vol. 52, no. 12, pp. 13 870–13 882, 2022.” is used as the comparison baseline for the semi-supervised manifold binary neural network method with optimal scaling proposed in the present invention.

[0116] In order to illustrate the stability of the SSMBNN-OS algorithm, the program was run several times under the same experimental conditions. The experimental comparison results are shown in Table 2, which compares the clustering performance of eleven clustering methods on five benchmark data sets, where the highest value is marked in bold and the second highest value is marked by an underline.

[0117] Table 2 Comparison of classification accuracy of eleven methods on five benchmark datasets

[0118]

[0119] As can be seen from Table 2, when learning and extracting deep features, the accuracy of the present invention on these datasets is excellent: the accuracy on the COIL20, MNIST, UMIST and AT&T datasets is the highest, which is better than the other ten classic baseline comparison methods; the accuracy on the COIL100 dataset is the second highest.

[0120] In order to prove that the model can achieve better performance with less labeled data, additional comparative experiments are conducted on the MNIST and COIL100 datasets. In the additional comparative experiments, the labeled data of COIL100 is reduced by 20%, and the labeled data of MNIST is reduced by 50%. The comparison results of the present invention and the other six methods are shown in Table 3, where the highest value is marked in bold and the second highest value is marked by underline.

[0121] Table 3 Comparison of classification accuracy of seven clustering methods on two benchmark datasets

[0122]

[0123] As can be seen from Table 3, in additional experiments, the designed SSMBNN-OS obtains the second highest accuracy on the COIL100 and MNIST datasets. When the COIL100 and MNIST labeled data are reduced by 20% and 50% respectively, the accuracy of the present invention is only reduced by 0.2% and 0.11% respectively, which verifies the superiority of the present invention in using less labeled data.

[0124] To demonstrate the superiority of the scale term, simulation experiments were conducted on COIL20 to compare SSMBNN with SSMBNN-OS. SSMBNN-OS uses a self-learning scale term to ensure the optimal scale for each iteration. The results are shown in Figure 2. Figure 3 shown.

[0125] from Figure 3 It can be seen that the scale term not only speeds up the convergence speed, but also significantly improves the classification accuracy, proving that the scale strategy is indeed effective.

[0126] from Figure 4It can be seen that the performance and convergence of the proposed method are indeed effective: SSMBNN-OS achieves convergence in only 5 iterations, while other operations require more iterations. This proves the fast convergence of SSMBNN-OS.

Claims

1. A method for constructing a semi-supervised manifold binary neural network under optimal scale, characterized in that: The steps include: Step 1: Data preprocessing; Split the dataset X into vector labels X l and contains n u X of unknown labels u , that is, X=[X l ,X u ], where n l +n u =n; Define the function as follows: Step 2: Get the final data; Q=sign(AX)=sign(A[X l ,X u ])=sign([AX l ,AX u ])=[Q l ,Q u ] (1) in A represents the mapping matrix, Q l represents the preprocessing result of labeled data, Q u Represents the preprocessing result of unknown label data; Step 3: Define the category indicator matrix; Category indicator matrix The details are as follows: Where C represents the number of classes; Step 4: Feature extraction OLSR regression model; Let the projection matrix W∈R m*c , W T W = I, and the OLSR model for feature extraction is as follows: Step 5: Orthogonal least squares regression model with optimal scale; Due to the orthogonality constraint, the subspace W in equation (4) has a fixed scale. In order to solve the problem of scale variation, the scale term α is introduced into the equation, so that the orthogonal least squares regression problem with the optimal scale is expressed as: Step 6: Define the loss of labeled data; For the labeled dataset Q l Using cross entropy loss, the loss is defined as: where Y i represents the true label value, Y i ' is the predicted label value, H(·) represents the cross entropy, and l represents the number of labeled data; Step 7: Define the loss of unlabeled data; for unlabeled data, use the same processing steps as labeled data to obtain the predicted label Y of the unlabeled data j '; At the same time, use OLSR to learn the projection transformation matrix and obtain the predicted label Y j "; For Q u , the loss is defined as: Where u represents the number of unlabeled data; Step 8: Optimize the total loss function; The total loss function is defined as: L=L(Q l )+β(t)L(Q u ) (7) Where β(t) is the coefficient that balances the two loss terms; the formula for β(t) is: Among them, T1 and T2 represent β t The hyperparameters of the changes, β f Represents β t The upper limit value of , t represents the number of model iterations.

2. The method for constructing a semi-supervised manifold binary neural network under optimal scale according to claim 1, characterized in that: In step 5, the specific steps for solving the orthogonal least squares regression problem are as follows: Step 5-1: Input: binary data matrix Q∈R m*n , label matrix Y l ; The number of unlabeled data n u , the number of classes C; Step 5-2: Initialization: Random orthogonal matrix L T L=LL T =I and α=1; Step 5-3: Repeat the following steps until the convergence condition is met: (1) W = L(:, 1:C) and H = L(:, (C+1):m); (2) Calculate Y u pass (3) Calculate Y by (4) Update α by (5) Calculate B by B = Q[Y, αQ T H]∈R m*n ; (6) Update L by In B, singular value decomposition is performed; Step 5-4: Output: projection matrix W, unknown label matrix Y u .

Images