A lightweight neural network approach for edge computing

By reducing the number of convolution kernels and the number of feature paths in the neural network and using efficient convolutional operations, a lightweight Mobile-former network model is built, which solves the problems of neural network parameters and calculation complexity in edge devices, and realizes the Faster reasoning and longer battery life.

CN118690817BActive Publication Date: 2024-11-26HUNAN QIANGSHI INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411162748.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-26
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The existing lightweight neural networks still have parameters and computational complexity problems when deployed on edge devices, and it is difficult to further optimize to meet the deep neural network learning needs of portable terminals.

Method used

By reducing the number of convolution kernels and the number of target feature paths, and combining more efficient convolutional operations, a more efficient neural network architecture is built, and a Mobile-former network model is adopted, including BiFormer module, Mobile module, Mobile-Former module, Former module, and Former module and Former-Mobile modules, which use Ghost convolution and PC convolution instead of traditional convolution, perform feature mapping and attention operations to reduce model parameters and computational complexity.

Benefits of technology

It significantly reduces model size and computing requirements, improves inference speed and battery life, and meets the real-time and multi-task deployment requirements of edge computing scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118690817B_ABST
    Figure CN118690817B_ABST
Patent Text Reader

Abstract

The present invention discloses a neural network lightweight method for edge computing, which relates to the field of neural network technology, and specifically includes: selecting an initial data set, inputting the initial data set into a Mobile-former network model for processing, and outputting a classification result; wherein the Mobile-former network model includes a BiFormer module, a Mobile module, a Mobile-Former module, a Former module, and a Former-Mobile module. The present invention reduces the computational complexity of the model, so that the edge device can complete the reasoning task faster and provide a faster response time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a neural network lightweight method for edge computing. Background Art

[0002] In recent years, the research on lightweight neural network architecture has been highly valued by scholars at home and abroad, and several representative research ideas have emerged. The research is mainly carried out from the following three aspects: (1) Manual modeling of lightweight neural networks. (2) Automatic neural network architecture design using neural network structure search (NAS); (3) Compressing neural network models. At present, there have been some progress in the research of lightweight neural network architecture: Google proposed a lightweight network architecture MobileNetV1 based on deep separable convolution to replace traditional convolution. MobileNet V2 proposed an inverted residual module based on V1. The main idea of ​​MobileNet V3 is neural architecture search (NAS) and hardware-friendly structure. Face++ uses the method of point group convolution core channel permutation to construct ShuffleNet V1, and later proposed a better ShuffleNet V2. MnasNet and NasNet use reinforcement learning to learn the structure search strategy from the neural network structure, so as to realize the automatic construction of lightweight neural networks on portable devices. Deep compression uses the Huffman coding method to model convolutional neural networks, and performs weight sharing and weight quantization on them; adaptive learning (AMC) is a model compression algorithm based on reinforcement learning, which improves the compression rate while ensuring network performance. Tencent AILab Machine Learning Center has successfully developed the world's first automated deep learning model compression framework - PocketFlow. The PocketFlow framework is mainly composed of two components, namely the model compression / acceleration algorithm component and the hyperparameter optimization component. The new optimization toolkit in TensorFlow Lite can reduce the model size and improve the execution speed by introducing post-training model quantization technology. Tsinghua University proposed a lightweight three-dimensional target detection method and device based on sparse convolution pruning, which can effectively reduce the computational complexity of sparse convolution and realize lightweight 3D target detection. Zhejiang Normal University proposed a lightweight target detection method and system based on the improvement of YOLOv8, which can improve the performance of small target detection while making lightweight improvements to YOLOv8 to meet end-to-end industrial deployment. The Nanjing Institute of Intelligent Technology of the Institute of Microelectronics of the Chinese Academy of Sciences has disclosed a face recognition method and system based on a lightweight network structure design. The provided face recognition method has fundamental advantages in the deployment of mobile devices and embedded devices.

[0003] However, the current artificial design ideas of lightweight neural networks mainly focus on how to improve the network computing efficiency. Existing deep convolutional neural networks usually use a large number of feature channels and convolution kernels to achieve good learning effects, but this method easily generates a large amount of redundant information. Although lightweight neural networks have matured, there is still room for improvement in the parameters and computational complexity of the models deployed on edge devices. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a neural network lightweight method for edge computing. By reducing the number of convolution kernels, reducing the number of paths of target features, and combining more efficient convolution operations, a more efficient neural network architecture is constructed. On the basis of retaining the performance of the neural network, the network parameters are greatly reduced, and the computational complexity is reduced, ultimately achieving deep neural network learning and application for portable terminals.

[0005] To achieve the above object, the present invention provides a neural network lightweight method for edge computing, comprising:

[0006] Select an initial data set, input the initial data set into the Mobile-former network model for processing, and output a classification result;

[0007] The Mobile-former network model includes a BiFormer module, a Mobile module, a Mobile-Former module, a Former module and a Former-Mobile module.

[0008] Preferably, the process of the BiFormer module processing the initial data set includes:

[0009] Obtaining a linear feature map of the initial data set, and calculating a correlation adjacency matrix between regions of different maps, and retaining the first k connections of each region, and saving the indexes of the first k connections row by row;

[0010] Aggregate the tensors of index vectors and value vectors to obtain K and V features, and use attention operation on the K and V features to obtain X features, and pass the X features to the Mobile module and the Mobile-Former module.

[0011] Preferably, the method for obtaining the linear feature map of the initial data set is:

[0012]

[0013] In the formula, is the number of processing times, is the weight matrix, forW different variants of Q To generate the query results, for W Index vector matrix operations, The X feature undergoes r matrix operations. is the first vector matrix feature, is the second vector matrix feature.

[0014] Preferably, the method for calculating the inter-region correlation adjacency matrix of different mappings is:

[0015]

[0016] In the formula, is the query vector matrix, is the number of processing times, is the key vector matrix, For transposition, is the attention score matrix.

[0017] Preferably, the method of saving the indexes of the first k connections row by row is:

[0018]

[0019] In the formula, To obtain the matrix The index of the maximum value, is the routing index matrix.

[0020] Preferably, the method of aggregating the tensors of the index vector and the value vector is:

[0021]

[0022] In the formula, From the characteristics According to the routing index matrix The results are retrieved from the specified location. From the characteristics According to the routing index matrix The results are retrieved from the specified location. is the first vector matrix feature, is the second vector matrix feature, For retrieval operations.

[0023] Preferably, the method for obtaining the X feature is:

[0024]

[0025] In the formula, is the self-attention mechanism, QTo generate the query results, is the cross entropy loss.

[0026] Preferably, the Mobile-Former module and the Mobile module in parallel represent the local feature map of the image processed by the BiFormer module as x, and randomly generate a global token as z. The input x and z are divided into H The multi-head attention of each head is used to obtain the output result based on the attention operation. The specific method is as follows:

[0027]

[0028] In the formula, is the output result obtained after the attention operation, is the self-attention mechanism, For the h The input vector of the attention head, For the h The query weight matrix of the attention head, For the h The key vector of the attention head, is the linear transformation weight matrix, h For the h The value vector of the attention heads;

[0029] Among them, the Mobile module uses Ghost convolution instead of 1*1 convolution, and uses PC convolution instead of depth-wise separable convolution.

[0030] Preferably, the output of the Mobile module is passed into the Former module, processed by the multi-head self-attention mechanism MHA and the feedforward network FFN, and normalized using the post layer to obtain the output Z.

[0031] Preferably, the Former-Mobile module processes X and Z output after being processed by the Mobile module and the Former module to obtain output, and the specific method is:

[0032]

[0033] In the formula, is the output result of the Former-Mobile module, is the self-attention mechanism, For the h The key vector of the head, For the h The input vector of the head, For the h The key weight matrix of the attention head,h For the h The value vector of the attention head, H is the number of attention heads, For the h The weight matrix of the values ​​of the attention heads.

[0034] Compared with the prior art, the present invention has the following advantages and technical effects:

[0035] The present invention significantly reduces the size of the model and can accommodate more models within a limited storage space, thereby allowing multiple tasks or models to be deployed on a device at the same time, thereby meeting the diverse needs of users.

[0036] The present invention significantly reduces the computational requirements of the model and improves the reasoning speed. In edge computing scenarios, real-time performance is a key indicator, especially for applications that require fast response. By reducing the computational complexity of the model, edge devices can complete reasoning tasks faster and provide faster response times.

[0037] The present invention can also reduce power consumption. In mobile devices and embedded systems, power consumption is an important consideration. High power consumption will cause the device to consume power too quickly, reduce user experience and increase charging frequency. This method can reduce the computational load of the model, thereby reducing the power consumption of the device, extending battery life, and improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 This is a flow chart of a neural network lightweight method for edge computing according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] This paper proposes a neural network lightweight method for edge computing, such as Figure 1 ,include:

[0043] Select the initial data set, input the initial data set into the Mobile-former network model for processing, and output the classification results;

[0044] Among them, the Mobile-former network model includes BiFormer module, Mobile module, Mobile-Former module, Former module and Former-Mobile module.

[0045] The present invention mainly optimizes the network structure based on the Mobile-former algorithm, replaces the stem module with the BiFormer module, and improves the Mobile module to improve the algorithm performance.

[0046] In this embodiment, CIFAR10 and CIFAR100 are used as training sets, and the images are input into the trained model to accurately identify and output the image classification results.

[0047] (1) BiFormer module

[0048] 1) Input a 224*224 size picture , whose image can be obtained by the following expression to obtain the linear mapping , , .

[0049] (1-1)

[0050] in, is the number of processing times, is the weight matrix, for W Different variants of , used to generate queries Q , for W Different variants of are used to generate the key K.

[0051] 2) Calculation and The adjacency matrix of the inter-region correlation is:

[0052] (1-2)

[0053] in, To query the vector matrix, After processing, we get is the key vector matrix, and After processing, we get is transposed;

[0054] 3) Keep the top k connections of each region to prune the dependency graph. Routing index matrix , saving the indexes of the first k connections row by row.

[0055] (1-3)

[0056] in, To obtain the matrix The index of the largest value.

[0057] 4) A tensor gathering index vectors and value vectors.

[0058] (1-4)

[0059] in, For According to the routing index matrix The results are retrieved from the specified location. For According to the routing index matrix The results are retrieved from the specified location. is a vector matrix, is a vector matrix, For retrieval operations;

[0060] 5) The aggregated K and V features are used for attention operation to obtain X features, which are then passed to the Mobile module and the Mobile-former module.

[0061] (1-5)

[0062] It is a self-attention mechanism that calculates the representation of each position in a sequence based on its relationship with other positions in the sequence. is the cross entropy loss.

[0063] (2) Mobile module

[0064] The local feature map of the image processed by Biformer is represented as x, and the output z of the parallel processed Former module is passed to the Mobile module, and the output is obtained through the following operations. Compared with the original model, this module uses Ghost convolution instead of 1*1 convolution and PC convolution instead of depthwise separable convolution, which greatly reduces the number of model parameters and calculations.

[0065] (3) Mobile-Former Module

[0066] The Mobile-Former module is run in parallel with the Mobile module. The local feature map of the image processed by the Biformer module is represented as x, and the randomly generated global token is represented as z. The specific process is:

[0067] The attention operation is expressed as follows.

[0068] The input x and z are split into H Multi-head attention of heads, using attention operation to get output.

[0069] (3-1)

[0070] In the formula, is the output result obtained after the attention operation, is the self-attention mechanism, For the h The input vector of the attention head, For the h The query weight matrix of the attention head, For the h The key vector of the attention head, is the linear transformation weight matrix, h For the h The value vector of the attention heads;

[0071] (4) Former module

[0072] The output of the Mobile module is passed to the Former module, processed by the Multi-Head Self-Attention Mechanism (MHA) and the Feed-Forward Network (FFN). The post (post layer normalization) layer is used for normalization. The output Z is obtained.

[0073] (5) Former-Mobile Module

[0074] The Former-Mobile module has the same architecture as the Mobile-Former module, but the input data is different. The outputs X and Z processed by the Mobile and Former modules are processed to obtain the output, specifically:

[0075] (5-1)

[0076] In the formula, is the output result of the Former-Mobile module, is the self-attention mechanism, For the h The key vector of the attention head, For the hThe input vector of the attention head, For the h The key weight matrix of the attention head, h For the h The value vector of the attention head, H is the number of attention heads, For the h The weight matrix of the values ​​of the attention heads.

[0077] The model of the present invention uses a mobile-former to achieve more flexible computing power allocation through double-layer routing while maintaining the lightweight of the network itself. First, the BiFormer module is used to replace the original stem part in the network. The BiFormer module is a visual Transformer model combined with Bi-level Routing Attention. The core idea of ​​the BiFormer module is to introduce a double-layer routing attention mechanism. In the BiFormer module, each image block is associated with a position router. These position routers assign image blocks to upper and lower routers according to specific rules. The upper router is responsible for capturing global context information, while the lower router is responsible for capturing details of local areas. Specifically, the upper router interacts with all image blocks through a global self-attention mechanism and generates a global image representation. The lower router uses a local self-attention mechanism to interact with each image block and its neighboring image blocks and generate a local image representation. Through this double-layer routing attention mechanism, the BiFormer module can capture global and local feature information at the same time, thereby improving the performance of the model in visual tasks.

[0078] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A neural network lightweight method for edge computing, characterized in that: include: Select an initial image data set, input the initial image data set into the Mobile-former network model for processing, and output an image classification result; The Mobile-former network model includes a BiFormer module, a Mobile module, a Mobile-Former module, a Former module and a Former-Mobile module; The initial image dataset is a 224*224 size image. , the method for obtaining the linear feature map of the initial image data set is: In the formula, is the number of processing times, is the weight matrix, are different variants of W, Q is the generated query result, is the index vector matrix operation of W, The X feature undergoes r matrix operations. is the first vector matrix feature, is a second vector matrix feature; the process of the BiFormer module processing the initial image data set includes: Obtaining a linear feature map of the initial image data set, and calculating a correlation adjacency matrix between regions of different maps, and retaining the first k connections of each region, and saving the indexes of the first k connections row by row; The method for calculating the inter-region correlation adjacency matrix of different mappings is: In the formula, is the query vector matrix, is the number of processing times, is the key vector matrix, For transposition, is the attention score matrix; Aggregate the tensors of index vectors and value vectors to obtain K and V features, and use attention operation on the K and V features to obtain X features, and pass the X features to the Mobile module and the Mobile-Former module; The Mobile-Former module and the Mobile module in parallel represent the local feature map of the image processed by the BiFormer module as x, and randomly generate a global token as z. The input x and z are divided into multi-head attention with H heads, and the output result is obtained based on the attention operation. The specific method is: In the formula, is the output result obtained after the attention operation, is the self-attention mechanism, is the input vector of the h-th attention head, is the query weight matrix of the h-th attention head, is the key vector of the h-th attention head, is the linear transformation weight matrix, h is the value vector of the h-th attention head; Among them, the Mobile module uses Ghost convolution instead of 1*1 convolution, and PC convolution instead of depth-separable convolution; The output of the Mobile module is passed into the Former module, and processed by the multi-head self-attention mechanism MHA and the feedforward network FFN, and normalized using the post layer to obtain the output Z; The Former-Mobile module processes X and Z output after being processed by the Mobile module and the Former module to obtain output, and the specific method is as follows: In the formula, is the output result of the Former-Mobile module, is the self-attention mechanism, is the key vector of the h-th head, is the input vector of the hth head, is the key weight matrix of the h-th attention head, h is the value vector of the h-th attention head, H is the number of attention heads, is the weight matrix of the values ​​of the h-th attention head.

2. The neural network lightweight method for edge computing according to claim 1 is characterized in that: The method to save the index of the first k connections row by row is: In the formula, To obtain the matrix The index of the maximum value, is the routing index matrix.

3. The neural network lightweight method for edge computing according to claim 2 is characterized in that: The method to aggregate the tensor of index vectors and value vectors is: In the formula, From the characteristics According to the routing index matrix The results are retrieved from the specified location. From the characteristics According to the routing index matrix The results are retrieved from the specified location. is the first vector matrix feature, is the second vector matrix feature, For retrieval operations.

4. The neural network lightweight method for edge computing according to claim 3 is characterized in that: The method for obtaining the X feature is: In the formula, is the self-attention mechanism, Q is the generated query result, is the cross entropy loss.

Citation Information

Patent Citations

  • Lightweight deep neural network training method, system, device and medium

    CN116187420A

  • Lightweight feature reservation human body action detection method based on graph neural network

    CN117373116A

  • Radar working mode identification method based on multi-scale visual Transform

    CN117746163A

  • Image recognition method and apparatus based on neural network model, and terminal device

    WO2024001123A1