Unlock instant, AI-driven research and patent intelligence for your innovation.

Node processing method and device based on parameter server, equipment and storage medium

A node processing and server technology, applied in the field of big data, can solve the problem of parameter server failure at the node and cannot be effectively processed, so as to avoid waste and ensure fault tolerance

Active Publication Date: 2020-05-12
GUANGZHOU WERIDE TECH LTD CO
View PDF10 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The main purpose of the present invention is to propose a parameter server-based node processing method, device, equipment and storage medium, aiming to solve the problem that the parameter server in the prior art cannot effectively handle when the node fails

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Node processing method and device based on parameter server, equipment and storage medium
  • Node processing method and device based on parameter server, equipment and storage medium
  • Node processing method and device based on parameter server, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0050] Such as figure 1 as shown, figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the solution of the embodiment of the present invention.

[0051] The node processing device based on the parameter server in this embodiment of the present invention may be a PC or a server device.

[0052] Such as figure 1As shown, the parameter server-based node processing device may include: a processor 1001 , such as a CPU, a network interface 1004 , a user interface 1003 , a memory 1005 , and a communication bus 1002 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 10...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the technical field of big data, and discloses a node processing method based on a parameter server, and the method comprises the steps: obtaining the global resource information of the parameter server so as to construct a dynamic global task chart; detecting whether a failure node exists in the global resource information or not based on the dynamic global task chart; ifyes, judging the operation task corresponding to the failure node as a failure task; and deploying the failure task to nodes except the failure node corresponding to the failure task. The invention further discloses a node processing device based on the parameter server, equipment and a storage medium. The dynamic global task chart is constructed, the information related to the second operation task is added to the chart, and when the failure node is found, the failure task can be deployed to the nodes except for the failure node corresponding to the failure task timely and effectively to ensure that the task is not interrupted so that the nodes except for the failure node can continue to complete the task.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a machine learning-based parameter server-based node processing method, device, equipment and storage medium. Background technique [0002] In cloud computing and data center environments, node failure is a common and inevitable problem, and node failure will cause the efficiency of distributed training tasks to be reduced or completely interrupted. Therefore, it is a very valuable research in the direction of distributed machine learning to make the system realize the fault tolerance for node failure to ensure the training efficiency is stable and not interrupted. [0003] Under the existing parameter server architecture, the fault tolerance mechanism is limited to fault tolerance at the level of the overall training task, that is, in the case of failure of a non-critical node, the scope of tasks assigned to other nodes is readjusted. Once a critical node fails, the ta...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08G06N20/00
CPCH04L67/1034H04L67/10G06N20/00
Inventor 聂鑫陈飞韩旭
Owner GUANGZHOU WERIDE TECH LTD CO