Data processing method, device, equipment and storage medium

Abstract

The application discloses a data processing method, device, equipment and storage medium, belonging to the technical field of artificial intelligence. In the embodiment of the present application, on the one hand, the devices in the data processing system are connected in a ring structure, and each device has two communication links with other devices. Communicating with other devices, the data processing system has good stability and robustness. On the other hand, when the data processing system performs data processing, the model parameters determined by each device are sequentially transmitted according to the above-mentioned communication link, and each device fuses the received model parameters with the model parameters determined by itself before transmitting, The amount of data transmitted between devices is small, and there is no need to send model parameters to one device collectively, which can effectively avoid the problems of overload and communication congestion, effectively improve the speed and efficiency of data processing, and ensure the stability of data processing.

Description

technical field [0001] The present application relates to the technical field of artificial intelligence, and in particular to a data processing method, device, equipment and storage medium. Background technique [0002] With the development of computer technology and the advancement of artificial intelligence (AI) technology, federated learning has gradually become a hot topic in the field of AI. Federated learning can complete the training tasks of machine learning models through multi-party collaboration. [0003] At present, multi-party collaborative federated learning adopts a customer service-server architecture. The federated learning system includes a central federated server, and other customer service devices (herein referred to as participants) in the federated learning system are connected to the central federated server for communication, forming star topology. When the federated learning system processes data, each participant usually trains the business proce...

AI Technical Summary

Problems solved by technology

[0004] This customer-server architecture (that is, star topology) is prone to single-point communication congestion. For example, when all participants send their model parameters to the central federated server, the central federated server is prone to overload and communication congestion, and then cannot receive The model parameters sent by the participants, causing the system to crash

Especially when the scale of the business processing model is large, this phenomenon is more serious

Moreover, in the star topology, there is only one communication link between a participant and the central federated server. It is easy for a participant to go offline due to a temporary communication link interruption and temporarily leave the federated learning system, interrupting the federated learning process

Therefore, the stability and robustness of the above data processing methods are very poor

Images