Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Combined learning method and device based on gradient momentum acceleration

A learning method and momentum technology, applied in the field of joint learning, which can solve problems such as improvement without considering algorithm convergence, slow algorithm convergence, etc.

Active Publication Date: 2020-03-17
ANHUI CHAOQING INFORMATION ENG
View PDF3 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The existing joint learning algorithm uses the GD (GradientDescent, gradient descent) algorithm to perform the local update step, without considering the improvement of the algorithm convergence caused by the previous weight change, and the algorithm converges slowly

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Combined learning method and device based on gradient momentum acceleration
  • Combined learning method and device based on gradient momentum acceleration
  • Combined learning method and device based on gradient momentum acceleration

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0049] like figure 1 Shown is a structural diagram of a joint learning method based on gradient momentum acceleration provided by the present invention. In the figure, the local learning model refers to the machine learning model embedded on each edge node, and the global learning model refers to the global momentum parameter The solution formula of d(t) and the global model parameter w(t), a joint learning method based on gradient momentum acceleration, the joint learning method adopts a distributed system and is applied to image recognition and speech recognition, and the distributed system Including several edge nodes and a central server connecting all edge nodes; the joint learning method includes:

[0050] Step 1: Embed the same machine learning model on each edge node, and execute the momentum gradient descent algorithm in the current aggregation interval to obtain the model parameters and momentum parameters at each moment in the current aggregation interval; the speci...

Embodiment 2

[0074] Corresponding to Embodiment 1 of the present invention, Embodiment 2 of the present invention provides a joint learning device based on gradient momentum acceleration. The joint learning device adopts a distributed system and is applied to image recognition and speech recognition. The distributed system Including several edge nodes and a central server connecting all edge nodes; the united learning device includes:

[0075] The parameter acquisition module is used to divide the training process into several aggregation intervals, each aggregation interval corresponds to the set duration; embed the same machine learning model on each edge node, and execute the momentum gradient descent algorithm in the current aggregation interval Obtain the model parameters and momentum parameters at each moment in the current aggregation interval;

[0076] The aggregation module is used for each edge node to simultaneously send the model parameters and momentum parameters to the centra...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a combined learning method and device based on gradient momentum acceleration, and the method comprises the steps: embedding the same machine learning model on each edge node,and executing a momentum gradient descent algorithm in a current aggregation interval to obtain a model parameter and a momentum parameter at each moment in the current aggregation interval; enablingthe central server to aggregate the model parameters to obtain global model parameters, and aggregate the momentum parameters to obtain global momentum parameters; substituting the global model parameters in the current aggregation interval into a loss function formula to obtain a loss function value, comparing the loss function value with the loss function value obtained in the previous aggregation interval to obtain optimized global model parameters, and obtaining the optimized global model parameters after calculation of all the aggregation intervals is completed. The method has the advantages that the momentum gradient descent algorithm is used in the local updating process of joint learning, namely the parameter updating process of the edge nodes, and the algorithm convergence speed is high.

Description

technical field [0001] The present invention relates to the field of joint learning, and more specifically relates to a joint learning method and device based on gradient momentum acceleration. Background technique [0002] FL (Federated Learning) is a distributed machine learning technology that can effectively use the limited computing and communication resources of edge nodes to train optimal model learning performance. The structure of FL includes a CS (Central Server, central server) and many ENs (EdgeNode, edge nodes). On EN, raw data is collected and stored in the storage unit of EN, and a machine learning model embedded in EN is used to train these local data, so EN does not need to send these local data to CS. Only the machine learning model parameters of the node are updated synchronously between CS and EN of FL, which we call Weight. This can not only reduce the amount of data communicated between nodes and servers, but also protect the privacy of user data (the...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06N20/00
CPCG06N20/00Y02T10/40
Inventor 卢青松汪思睿王培青金磊魏洪伟
Owner ANHUI CHAOQING INFORMATION ENG
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com