Model training method and communication apparatus

Abstract

A model training method and a communication apparatus. The method comprises: a communication apparatus receiving first information, which is configured to indicate an input dataset of an l-th layer of an intelligent model and a gradient accumulation factor set corresponding to the l-th layer, wherein l is a positive integer; and according to the input dataset of the l-th layer and the gradient accumulation factor set corresponding to the l-th layer, the communication apparatus determining a gradient set corresponding to the l-th layer, wherein one gradient accumulation factor in the gradient accumulation factor set is configured to determine a plurality of gradients in the gradient set, and the gradient set is configured to determine the intelligent model. The transmission overhead can be reduced, and the resource utilization rate can be increased.

Description

Model training method and communication device Technical Field

[0001] The present application relates to the field of communications, and more specifically, to a model training method and a communication device. Background Art

[0002] With the continuous development of computing power, algorithms, and data-related technologies, the three driving forces behind artificial intelligence (AI) and machine learning (ML), AI / ML technologies have ushered in a new technological revolution in human society. AI / ML technologies have significant application potential in complex and unknown environment modeling and learning, and are applied to channel prediction, intelligent signal generation and processing, network status tracking and intelligent scheduling, and network optimization and deployment. They are expected to promote the evolution of future communication paradigms and transform network architectures, and are of great significance and value to the research of future mobile communication technologies.

[0003] The training process of an AI model can be completed by multiple devices working together. For example, in distributed federated learning, multiple terminals can independently train AI models using local sample data and send the trained model weights or gradients to a central server. The central server aggregates the received model weights or gradients and sends the aggregated results to multiple terminals so that the terminals can perform the next model training. The above process is repeated until the model converges. This can solve the time-consuming problem of collecting data when a single device independently performs model training, realize the sharing of device resources (such as data and processing power, etc.), increase the diversity of training data, and improve training performance.

[0004] However, during the distributed model training process, model weights / gradients are transferred between devices, resulting in huge data transmission overhead. Especially when applied in wireless communication networks, the occupation of air interface resources may affect the normal communication services of the terminal.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a model training method and a communication device, which can reduce transmission overhead and improve resource utilization.

[0007] In a first aspect, a model training method is provided. The method can be performed by a communication device or a device configured in (or used for) a communication device (such as a chip, chip system, or processor). The method can also be implemented by a logical node, logic module, or software that can implement all or part of the communication device. The following description uses the second communication device performing the method as an example.

[0008] The method includes: a second communication device receives first information indicating an input data set of layer l of an intelligent model and a set of gradient accumulation factors corresponding to layer l, where l is a positive integer. The second communication device determines a gradient set corresponding to layer l based on the input data set of layer l and the set of gradient accumulation factors corresponding to layer l. A gradient accumulation factor in the set of gradient accumulation factors is used to determine multiple gradients in the gradient set, and the gradient set is used to determine the intelligent model.

[0009] According to the above scheme, multiple communication devices jointly perform model training. When the gradient set of a neuron layer needs to be transmitted between communication devices, the input data set and the corresponding gradient accumulation factor set obtained by decomposing the gradient set corresponding to the neuron layer can be transmitted. The receiving device can determine the gradient set based on the input data set and the gradient accumulation factor set. The combination of each gradient accumulation factor in the gradient accumulation factor set and the input data set can determine multiple gradients in the gradient set. Compared with the method of directly transmitting the gradient set, each training can reduce transmission overhead and improve resource utilization. The model training process that includes multiple iterative training can greatly reduce transmission overhead and improve resource utilization.

[0010] In conjunction with the first aspect, in certain embodiments of the first aspect, the method further includes: determining, by a second communication device, a weight set for the intelligent model based on the gradient set, the gradient set including a gradient corresponding to each weight in the weight set; and determining, by the second communication device, an updated intelligent model based on the weight set.

[0011] According to the above solution, the second communication device can determine the weight set of the intelligent model based on the gradient set, thereby determining the updated intelligent model obtained after this iterative training.

[0012] In conjunction with the first aspect, in certain embodiments of the first aspect, the first layer is a fully connected layer, the first layer includes N nodes, and the l+1th layer of the intelligent model includes M nodes, where N and M are positive integers greater than 1. The input data set of the first layer includes N input data corresponding to the N nodes, and the gradient accumulation factor set corresponding to the first layer includes M gradient accumulation factors corresponding to the M nodes, wherein the gradient set includes T gradients, where T is the product of N and M.

[0013] According to the above scheme, for the fully connected layer, if the first communication device directly sends the gradient set to the second communication device, M×N parameters (i.e., T parameters) need to be transmitted. When M and N are greater than 2, M+N is less than M×N, which can reduce the transmission overhead of the model parameters. The first communication device can use this method to transmit model parameters for each layer of the intelligent model, especially when each layer contains a large number of nodes and a large number of training times, which can further reduce the transmission overhead.

[0014] In combination with the first aspect, in certain embodiments of the first aspect, the second communication device determines the gradient set corresponding to the lth layer based on the input data set of the lth layer and the gradient accumulation factor set corresponding to the lth layer, including: the second communication device determines T gradients in the gradient set based on the input data of each node in the N nodes and the gradient accumulation factor corresponding to each node in the M nodes, and one gradient g among the T gradients mn It is the product of the input data of the nth node among the N nodes and the gradient accumulation factor corresponding to the mth node among the M nodes, where n is a positive integer less than or equal to N, and m is a positive integer less than or equal to M.

[0015] In conjunction with the first aspect, in certain embodiments of the first aspect, the input data set of the first layer includes input data of P channels. The first layer includes Q convolution kernels, the Q convolution kernels corresponding to the Q channels of the output data of the first layer, and the gradient accumulation factor set corresponding to the first layer includes Q gradient accumulation factor subsets corresponding to the Q convolution kernels. The gradient set includes R gradient subsets, the R gradient subsets including P gradient subsets corresponding to each of the Q convolution kernels, and the P gradient subsets corresponding to each convolution kernel correspond to the P channels, where P and Q are positive integers, and R is the product of Q and P.

[0016] According to the above solution, for the convolution layer, the gradient accumulation factor set includes Q gradient accumulation factor subsets, and the Q gradient accumulation factors correspond to Q convolution kernels one by one. Each gradient accumulation factor subset is combined with the input data set to determine the gradients corresponding to the P channels corresponding to the corresponding convolution kernel. The first communication device transmits the gradient accumulation factor set δ obtained by the gradient decomposition corresponding to the lth layer. l and input data set a l-1 Instead of transmitting the gradient set g l , which can reduce the transmission overhead of model parameters. The first communication device can use this method to transmit model parameters for each layer of the intelligent model. Especially when the dimension of the input data is small and the number of training times is large, the transmission overhead can be reduced to a greater extent.

[0017] In combination with the first aspect, in certain embodiments of the first aspect, the second communication device determines a gradient set based on the input data set of the lth layer and the gradient accumulation factor set corresponding to the lth layer, including: the second communication device uses a convolution operation to determine R gradient subsets in the gradient set based on the input data of each channel in the P channels and the gradient accumulation factor subset corresponding to each convolution kernel in the Q convolution kernels, and a gradient subset G in the R gradient subsets. qp It is obtained by using a convolution operation based on the gradient accumulation factor corresponding to the p-th channel in the P channels and the q-th convolution kernel in the Q convolution kernels.

[0018] In combination with the first aspect, in certain embodiments of the first aspect, the method further includes: a second communication device sending the gradient set, where the gradient set is used to determine the intelligent model.

[0019] According to the above scheme, after the second communication device determines the gradient set, it can send the gradient set to the first communication device, so that the first communication device determines the weight set based on the gradient set and thus determines the updated intelligent model, and performs the next model training based on the updated intelligent model, or the second communication device can send the weight set obtained based on the gradient set to the second communication device, so that the first communication device determines the updated intelligent model based on the weight set and thus performs the next model training.

[0020] In a second aspect, a model training method is provided. This method can be performed by a communication device or a device configured in (or used for) a communication device (such as a chip, chip system, or processor). It can also be implemented by a logical node, logic module, or software that can implement all or part of the communication device. The following description uses the first communication device executing this method as an example.

[0021] The method includes: a first communication device executing a model training process for an intelligent model to determine an input data set for layer l of the intelligent model and a set of gradient accumulation factors corresponding to the layer l, where l is a positive integer. The first communication device transmits first information indicating the input data set for the layer l and the set of gradient accumulation factors corresponding to the layer l. A gradient accumulation factor in the set of gradient accumulation factors is used to determine multiple gradients in a gradient set corresponding to the layer l.

[0022] In combination with the second aspect, in certain embodiments of the second aspect, the first communication device determines the gradient accumulation factor set corresponding to the lth layer of the intelligent model, including: the first communication device determines the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer.

[0023] In combination with the second aspect, in certain embodiments of the second aspect, the intelligent model includes L layers, l is less than or equal to L, and the first communication device determines the gradient accumulation factor set corresponding to the lth layer of the intelligent model, including: if l is equal to L, the first communication device determines the gradient accumulation factor set corresponding to the Lth layer based on the output data set and label set of the Lth layer, and the label set is used to determine the weight set of the intelligent model.

[0024] In conjunction with the second aspect, in certain embodiments of the second aspect, the first layer is a fully connected layer, the first layer includes N nodes, and the l+1th layer of the intelligent model includes M nodes, where N and M are positive integers greater than 1. The input data set of the first layer includes N input data corresponding to the N nodes, and the gradient accumulation factor set corresponding to the first layer includes M gradient accumulation factors corresponding to the M nodes, wherein the gradient set includes T gradients, where T is the product of N and M.

[0025] In combination with the second aspect, in certain embodiments of the second aspect, the first communication device determines the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer, including: the first communication device determines the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer, using partial derivative operations and / or multiplication operations.

[0026] In conjunction with the second aspect, in certain embodiments of the second aspect, the input data set of the first layer includes input data of P channels. The first layer includes Q convolution kernels, the Q convolution kernels corresponding to the Q channels of the output data of the first layer, and the gradient accumulation factor set corresponding to the first layer includes Q gradient accumulation factor subsets corresponding to the Q convolution kernels. The gradient set includes R gradient subsets, the R gradient subsets including P gradient subsets corresponding to each of the Q convolution kernels, and the P gradient subsets corresponding to each convolution kernel correspond to the P channels, where P and Q are positive integers, and R is the product of Q and P.

[0027] In combination with the second aspect, in certain embodiments of the second aspect, the first communication device determines the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer, including: the first communication device determines the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer, using partial derivative operations and / or convolution operations.

[0028] In a third aspect, a communication device is provided. In one design, the device may include a module corresponding to the method / operation / step / action described in the first aspect or any one of the embodiments of the first aspect. The module may be a hardware circuit, software, or a combination of a hardware circuit and software. In one design, the device includes a transceiver unit for receiving first information, where the first information is used to indicate the input data set of the lth layer of the intelligent model and the gradient accumulation factor set corresponding to the lth layer, where l is a positive integer. A processing unit is used to determine the gradient set corresponding to the lth layer based on the input data set of the lth layer and the gradient accumulation factor set corresponding to the lth layer, wherein a gradient accumulation factor in the gradient accumulation factor set is used to determine multiple gradients in the gradient set, and the gradient set is used to determine the intelligent model.

[0029] In conjunction with the third aspect, in certain embodiments of the third aspect, the processing unit is further configured to determine a weight set of the intelligent model based on the gradient set, where the gradient set includes a gradient corresponding to each weight in the weight set. Furthermore, the processing unit is further configured to determine an updated intelligent model based on the weight set.

[0030] In conjunction with the third aspect, in certain embodiments of the third aspect, the first layer is a fully connected layer, the first layer includes N nodes, the l+1th layer of the intelligent model includes M nodes, and N and M are positive integers greater than 1;

[0031] The input data set of the lth layer includes N input data corresponding to the N nodes, and the gradient accumulation factor set corresponding to the lth layer includes M gradient accumulation factors corresponding to the M nodes, wherein the gradient set includes T gradients, and T is the product of N and M.

[0032] In conjunction with the third aspect, in certain embodiments of the third aspect, the processing unit is specifically configured to determine, based on the input data of each node in the N nodes and the gradient accumulation factor corresponding to each node in the M nodes, T gradients in the gradient set, one gradient g among the T gradients mn It is the product of the input data of the nth node among the N nodes and the gradient accumulation factor corresponding to the mth node among the M nodes, where n is a positive integer less than or equal to N, and m is a positive integer less than or equal to M.

[0033] In conjunction with the third aspect, in certain embodiments of the third aspect, the input data set of the first layer includes input data of P channels. The first layer includes Q convolution kernels, the Q convolution kernels corresponding to the Q channels of the output data of the first layer, and the gradient accumulation factor set corresponding to the first layer includes Q gradient accumulation factor subsets corresponding to the Q convolution kernels. The gradient set includes R gradient subsets, the R gradient subsets including P gradient subsets corresponding to each of the Q convolution kernels, and the P gradient subsets corresponding to each convolution kernel correspond to the P channels, where P and Q are positive integers, and R is the product of Q and P.

[0034] In combination with the third aspect, in certain embodiments of the third aspect, the processing unit is specifically configured to determine R gradient subsets in the gradient set using a convolution operation based on the input data of each channel in the P channels and the gradient accumulation factor subset corresponding to each convolution kernel in the Q convolution kernels, and a gradient subset G in the R gradient subsets. qp It is obtained by using a convolution operation based on the gradient accumulation factor corresponding to the p-th channel in the P channels and the q-th convolution kernel in the Q convolution kernels.

[0035] In combination with the third aspect, in certain embodiments of the third aspect, the transceiver unit is further used to send the gradient set, which is used to determine the intelligent model.

[0036] In a fourth aspect, a communication device is provided. In one design, the device may include a module corresponding to executing the method / operation / step / action described in the second aspect or any embodiment of the second aspect. The module may be a hardware circuit, software, or a combination of hardware circuit and software. In one design, the device includes a processing unit for executing a model training process of an intelligent model, determining an input data set of the lth layer of the intelligent model and a set of gradient accumulation factors corresponding to the lth layer, where l is a positive integer. A transceiver unit is configured to send first information, the first information being used to indicate the input data set of the lth layer and the set of gradient accumulation factors corresponding to the lth layer, wherein a gradient accumulation factor in the set of gradient accumulation factors is used to determine multiple gradients in the gradient set corresponding to the lth layer.

[0037] In combination with the fourth aspect, in certain embodiments of the fourth aspect, the processing unit is specifically used to determine the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer.

[0038] In combination with the fourth aspect, in certain embodiments of the fourth aspect, the intelligent model includes L layers, l is less than or equal to L, if l is equal to L, the processing unit is specifically used to determine the gradient accumulation factor set corresponding to the Lth layer based on the output data set and label set of the Lth layer, and the label set is used to determine the weight set of the intelligent model.

[0039] In conjunction with the fourth aspect, in certain embodiments of the fourth aspect, the first layer is a fully connected layer, the first layer includes N nodes, and the l+1th layer of the intelligent model includes M nodes, where N and M are positive integers greater than 1. The input data set of the first layer includes N input data corresponding to the N nodes, and the gradient accumulation factor set corresponding to the first layer includes M gradient accumulation factors corresponding to the M nodes, wherein the gradient set includes T gradients, where T is the product of N and M.

[0040] In combination with the fourth aspect, in certain embodiments of the fourth aspect, the processing unit is specifically used to determine the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer, using partial derivative operations and / or multiplication operations.

[0041] In conjunction with the fourth aspect, in certain embodiments of the fourth aspect, the input data set of the first layer includes input data of P channels. The first layer includes Q convolution kernels, the Q convolution kernels corresponding to the Q channels of the output data of the first layer, and the gradient accumulation factor set corresponding to the first layer includes Q gradient accumulation factor subsets corresponding to the Q convolution kernels. The gradient set includes R gradient subsets, the R gradient subsets including P gradient subsets corresponding to each of the Q convolution kernels, and the P gradient subsets corresponding to each convolution kernel correspond to the P channels, where P and Q are positive integers, and R is the product of Q and P.

[0042] In combination with the fourth aspect, in certain embodiments of the fourth aspect, the processing unit is specifically used to determine the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer, using partial derivative operations and / or convolution operations.

[0043] In a fifth aspect, a communication device is provided, comprising a processor. The processor is coupled to the memory, the memory is configured to store instructions, and the processor is configured to execute the instructions in the memory to cause the communication device to implement the method of the first aspect or the second aspect, and any possible implementation of the first aspect or the second aspect.

[0044] Optionally, the communication device further includes a memory, which can be used to execute instructions in the memory to implement the method in the above-mentioned first aspect or second aspect and any possible implementation manner of the first aspect or second aspect.

[0045] Optionally, the communication device further comprises a communication interface, and the processor is coupled to the communication interface. In the embodiment of the present application, the communication interface can be a transceiver, a pin, a circuit, a bus, a module, or other types of communication interfaces, without limitation.

[0046] In one implementation, the communication apparatus is a communication device (such as a terminal or a network device). When the communication apparatus is a communication device, the communication interface may be a transceiver, or an input / output interface.

[0047] In another implementation, the communication device is a chip configured in a communication device. When the communication device is a chip configured in a communication device, the communication interface may be an input / output interface.

[0048] Optionally, the transceiver may be a transceiver circuit. Optionally, the input / output interface may be an input / output circuit.

[0049] In a sixth aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is configured to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method of the first aspect or the second aspect, and any possible implementation of the first aspect or the second aspect.

[0050] In a specific implementation, the processor may be one or more chips, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be a transistor, a gate circuit, a trigger, or various logic circuits. The input signal received by the input circuit may be, for example, but not limited to, received and input by a receiver, and the signal output by the output circuit may be, for example, but not limited to, output to and transmitted by a transmitter. The input circuit and the output circuit may be the same circuit, which functions as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation of the processor and various circuits.

[0051] In the seventh aspect, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute the method in the above-mentioned first aspect or second aspect and any possible implementation of the first aspect or second aspect.

[0052] In an eighth aspect, a computer-readable storage medium is provided, which stores a computer program (also referred to as code, or instructions) which, when run on a computer, enables the computer to execute the method in the above-mentioned first aspect or second aspect and any possible implementation of the first aspect or second aspect.

[0053] In a ninth aspect, a communication system is provided, comprising at least one communication device provided in the third aspect and at least one communication device provided in the fourth aspect.

[0054] In the tenth aspect, a chip is provided, comprising at least one processor and a communication interface, wherein the communication interface is used to receive signals input into the chip or signals output from the chip, and the processor communicates with the communication interface and implements the method in the above-mentioned first aspect or second aspect and any possible implementation of the first aspect or second aspect through logic circuits or execution code instructions.

[0055] It can be understood that for the beneficial effects of the features corresponding to the first aspect in the second to tenth aspects, please refer to the relevant description in the first aspect and no repetition will be given. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] FIG1 is a schematic block diagram of a communication system applicable to an embodiment of the present application;

[0057] FIG1A is another schematic block diagram of a communication system applicable to an embodiment of the present application;

[0058] FIG2 is a schematic diagram of a fully connected layer of a neural network provided by the present application;

[0059] FIG3 is a schematic diagram of a convolutional layer of a neural network provided by the present application;

[0060] FIG4 is a schematic flow chart of the model training process provided in this application;

[0061] FIG5 is a schematic diagram of the joint model training provided by the present application;

[0062] FIG6 is a schematic flow chart of the model training method provided in this application;

[0063] FIG6A is a schematic diagram of an intelligent model including a fully connected layer provided by the present application;

[0064] FIG7 is another schematic flow chart of the model training method provided in this application;

[0065] FIG8 is a schematic structural diagram of the communication device of the present application;

[0066] FIG9 is another schematic structural diagram of the communication device of the present application;

[0067] FIG10 is another schematic structural diagram of the communication device of the present application. DETAILED DESCRIPTION

[0068] The technical solution in this application will be described below with reference to the accompanying drawings.

[0069] In the embodiments of this application, " / " can indicate that the objects associated with each other are in an "or" relationship. For example, A / B can mean A or B. "And / or" can be used to describe the existence of three relationships between the associated objects. For example, "A and / or B" can mean: A exists alone, A and B exists simultaneously, and B exists alone. A and B can be singular or plural. To facilitate the description of the technical solutions of the embodiments of this application, the words "first" and "second" can be used to distinguish them in the embodiments of this application. The words "first" and "second" do not limit the quantity or order of execution, and the words "first" and "second" do not necessarily mean different. In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. The use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete way to facilitate understanding. In the embodiments of the present application, at least one (kind) can also be described as one (kind) or multiple (kinds), and multiple (kinds) can be two (kinds), three (kinds), four (kinds) or more (kinds), and this application does not limit it.

[0070] Figure 1 is a schematic diagram of the architecture of a communication system 1000 to which the present application can be applied. As shown in Figure 1 , the communication system includes a radio access network (RAN) 100 and a core network (CN) 200. Optionally, the communication system 1000 may also include the Internet 300. The radio access network 100 may include at least one access network node (such as 110a and 110b in Figure 1 ) and at least one terminal (such as 120a-120j in Figure 1 ). The terminal is connected to the access network node via wireless communication. The access network node is connected to the core network via wireless communication or wired communication. The core network device and the access network node may be independent and distinct physical devices; or they may be the same physical device that integrates the functions of the core network device and the functions of the access network node; or other possible scenarios may exist, such as integrating the functions of the access network node and some of the core network device functions on one physical device, while another physical device implements the remaining functions of the core network device. This application does not limit the physical form of the core network device and the access network node. Terminals may be connected to each other via wired or wireless communication. The access network nodes can be connected to each other via wired or wireless means. Figure 1 is only a schematic diagram, and the communication system may also include other network devices, such as wireless relay devices and wireless backhaul devices.

[0071] An access network node may be an access network device, such as a base station, a Node B, an evolved Node B (eNodeB or eNB), a transmission reception point (TRP), a next-generation Node B (gNB) in a fifth-generation (5G) mobile communication system, an access network node in an open radio access network (O-RAN or open RAN), a next-generation base station in a sixth-generation (6G) mobile communication system, a base station in a future mobile communication system, or an access node in a wireless fidelity (WiFi) system. Alternatively, an access network node may be a module or unit that performs some of the functions of a base station, for example, a centralized unit (CU), a distributed unit (DU), a centralized unit control plane (CU-CP) module, or a centralized unit user plane (CU-UP) module. The access network node can be a macro base station (such as 110a in Figure 1), a micro base station or an indoor station (such as 110b in Figure 1), a relay node or a donor node, etc. The specific technology and specific device form used by the access network node are not limited in this application. Among them, the 5G system can also be called a new radio (NR) system.

[0072] In the present application, the device for implementing the function of the access network node may be an access network node; or it may be a device capable of supporting the access network node to implement the function, such as a chip system, a hardware circuit, a software module, or a hardware circuit plus a software module, which device may be installed in the access network node or may be used in conjunction with the access network node. In the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices. For ease of description, the following description will take the case where the device for implementing the function of the access network node is an access network node, and optionally the case where the access network node is a base station as an example to describe the technical solution provided by the present application.

[0073] A terminal may also be referred to as a terminal device, user equipment (UE), mobile station, mobile terminal, etc. The terminal can be widely used in various scenarios for communication. The scenario includes, but is not limited to, at least one of the following scenarios: enhanced mobile broadband (eMBB), ultra-reliable low-latency communication (URLLC), massive machine-type communications (mMTC), device-to-device (D2D), vehicle-to-everything (V2X), machine-type communication (MTC), Internet of Things (IoT), virtual reality, augmented reality, industrial control, autonomous driving, telemedicine, smart grid, smart furniture, smart office, smart wearable, smart transportation, or smart city. The terminal can be a mobile phone, tablet computer, computer with wireless transceiver function, wearable device, vehicle, drone, helicopter, airplane, ship, robot, robotic arm, or smart home device, etc. This application does not limit the specific technology and specific device form adopted by the terminal.

[0074] In this application, a device for implementing a terminal function may be a terminal; it may also be a device capable of supporting the terminal in implementing the function, such as a chip system, a hardware circuit, a software module, or a hardware circuit plus a software module. The device may be installed in the terminal or used in conjunction with the terminal. For ease of description, the following describes the technical solution provided in this application using the example of the device for implementing the terminal function being a terminal, and optionally using the example of the terminal being a UE.

[0075] The base station and / or terminal can be fixed or movable. The base station and / or terminal can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; or can be deployed on the water surface; or can be deployed on aircraft, balloons and artificial satellites in the air. This application does not limit the environment / scenario in which the base station and terminal are located. The base station and the terminal can be deployed in the same or different environments / scenarios, for example, the base station and the terminal are deployed on land at the same time; or the base station is deployed on land and the terminal is deployed on water surface, etc., and no further examples are given.

[0076] The roles of base stations and terminals can be relative. For example, the helicopter or drone 120i in Figure 1 can be configured as a mobile base station. For terminals 120j accessing the wireless access network 100 via 120i, terminal 120i is a base station. However, for base station 110a, 120i can be a terminal, meaning that communication between 110a and 120i can occur via a wireless air interface protocol. Alternatively, communication between 110a and 120i can occur via a base station-to-base station interface protocol. In this case, 120i is also a base station relative to 110a. Therefore, base stations and terminals can be collectively referred to as communication devices (or communication equipment). 110a and 110b in Figure 1 can be referred to as communication devices with base station functionality, while 120a-120j in Figure 1 can be referred to as communication devices with terminal functionality.

[0077] Optionally, the protocol layer structure between the access network node and the terminal may include an artificial intelligence (AI) layer for transmitting data related to AI functions.

[0078] In the present application, an independent network element (e.g., an AI entity, an AI network element, an AI node, or an AI device) may be introduced into the communication system shown in FIG. 1 to implement AI-related operations. The AI ​​network element may be directly connected to the base station, or may be indirectly connected to the base station through a third-party network element. The AI ​​network element may also be an independently set network element in the communication system, for example, a core network element such as an access and mobility management function (AMF) network element or a user plane function (UPF) network element. Alternatively, the AI ​​network element may be a network element built into the communication system, or an AI entity (or an AI module or other name) may be configured within the network element in the communication system to implement AI-related operations. Optionally, the network element configured with the AI ​​entity may be a base station, a core network device, or a network management (OAM). OAM is used to operate, manage, and / or maintain core network devices (network management of core network devices) and / or to operate, manage, and / or maintain access network nodes (network management of access network nodes).

[0079] Optionally, in order to match and support AI, the terminal or terminal chip may include an AI entity to implement AI-related functions.

[0080] Optionally, in this application, the AI ​​entity can also be called other names, such as AI module or AI unit, etc., which are mainly used to implement AI functions (or called AI-related operations). This application does not limit its specific name.

[0081] In this application, the intelligent model may be referred to as an AI model, and the AI ​​model is a specific method for implementing AI functions. The AI ​​model characterizes the mapping relationship between the input and output of the model. The AI ​​model may be a neural network, a linear regression model, a decision tree model, a clustering (singular value decomposition, SVD) model, or other machine learning model. Among them, the AI ​​model may be referred to as an intelligent model, model, or other name without limitation. AI-related operations may include at least one of the following: data collection, model training, model information release, model testing (or model verification), model inference (or model reasoning, inference, or prediction, etc.), or inference result release, etc.

[0082] Figure 1A is another architectural diagram of a communication system to which the present application can be applied. As shown in Figure 1A, network elements in the communication system can be connected through interfaces. For example, the core network node and the access network node shown in Figure 1A can be connected through an NG interface, and the access network nodes can be connected through an Xn interface. The network elements in the communication system can also be connected through an air interface, such as the access network node and the terminal can be connected through an air interface. One or more AI modules are provided in one or more nodes of these network element nodes, such as the core network node, the access network node (RAN node), the terminal or the OAM (for clarity, only one AI module is shown in each node in Figure 1A). The access network node shown in Figure 1 can be a separate RAN node, or it can include multiple RAN nodes, for example, including a CU and a DU. The CU and / or DU can also be provided with one or more AI modules. Optionally, the CU can also be split into a CU-CP and a CU-UP. One or more AI models are provided in the CU-CP and / or CU-UP.

[0083] The AI ​​module is used to implement the corresponding AI function. The AI ​​modules deployed in different network elements can be the same or different. The model of the AI ​​module can implement different functions according to different parameter configurations. The model of the AI ​​module can be configured based on one or more of the following parameters: structural parameters (such as the number of neural network layers, the width of the neural network, the connection relationship between layers, the weight of the neuron, the activation function of the neuron, or at least one of the bias in the activation function), input parameters (such as the type of input parameters and / or the dimension of the input parameters), or output parameters (such as the type of output parameters and / or the dimension of the output parameters). Among them, the bias in the activation function can also be called the bias of the neural network.

[0084] An AI module can have one or more models. A model can infer an output, which includes one or more parameters. The learning, training, or inference processes of different models can be deployed on different nodes or devices, or on the same node or device.

[0085] The communication devices (such as the first communication device and / or the second communication device) provided in the embodiments of the present application may be network devices, such as the access network nodes, servers, or base stations described above. The communication devices (such as the first communication device and / or the second communication device) used in the embodiments of the present application may also be the terminals described above.

[0086] The solutions provided in the embodiments of the present application can be applied to various application scenarios, such as virtual reality (VR), augmented reality (AR), industrial control, industrial wireless sensor networks (IWSN), unmanned driving, telemedicine, smart grids, smart cities, video surveillance, smart retail, and / or smart homes.

[0087] Before describing the method of this application, some relevant knowledge about artificial intelligence is briefly introduced.

[0088] 1. Artificial Intelligence (AI)

[0089] Artificial intelligence (AI) is the ability for machines to learn and accumulate experience, enabling them to solve problems that humans can solve through experience, such as natural language understanding, image recognition, and / or chess. For example, machines can use computer hardware and software to simulate certain intelligent human behaviors. To achieve AI, machine learning (ML) or other methods can be used, and this application does not limit these methods.

[0090] 2. Training or learning

[0091] Training refers to the process of processing an AI model (or training model) to optimize the weighted values ​​(called weights) in the model so that the model learns to perform a specific task.

[0092] AI model training methods include, but are not limited to, supervised learning, unsupervised learning, reinforcement learning, and transfer learning. Unsupervised learning can also be called unsupervised learning.

[0093] Supervised learning uses machine learning algorithms to learn the mapping relationship between sample values ​​and sample labels based on collected sample values ​​and sample labels, and then expresses this learned mapping relationship using an AI model. The process of training a machine learning model is the process of learning this mapping relationship. During training, sample values ​​are input into the model to obtain the model's predicted values. The model parameters are optimized by calculating the error between the model's predicted values ​​and the sample labels (ideal values). Once the mapping relationship is learned, the learned mapping can be used to predict new sample labels. The mapping relationship learned by supervised learning can include linear mappings or nonlinear mappings. Based on the type of label, the learning task can be divided into classification tasks and regression tasks.

[0094] Unsupervised learning uses algorithms to discover inherent patterns in collected sample values. One type of unsupervised learning algorithm uses the samples themselves as supervisory signals, meaning the model learns the mapping from one sample to another. This is called self-supervised learning. During training, the model parameters are optimized by calculating the error between the model's predictions and the samples themselves. Self-supervised learning can be used in signal compression and decompression recovery applications. Common algorithms include autoencoders and generative adversarial networks.

[0095] Reinforcement learning, unlike supervised learning, is a type of algorithm that learns problem-solving strategies through interaction with the environment. Unlike supervised and unsupervised learning, reinforcement learning problems lack explicit label data for "correct" actions. Instead, the algorithm must interact with the environment to obtain reward signals from the environment, and then adjust its decision-making actions to maximize the reward signal value. For example, in downlink power control, the reinforcement learning model adjusts the downlink transmit power of each user based on the overall system throughput fed back by the wireless network, hoping to achieve higher system throughput. The goal of reinforcement learning is also to learn the mapping between environmental states and optimal decision-making actions. However, because the labels for "correct actions" are not available in advance, network optimization cannot be achieved by calculating the error between actions and "correct actions." Reinforcement learning training is achieved through iterative interaction with the environment.

[0096] During AI model training, a loss function can be defined. This function describes the gap or discrepancy between the AI ​​model's output and the ideal target value. Training an AI model involves adjusting the model's weights to keep the loss function below a threshold or meet the target requirement.

[0097] 3. Reasoning

[0098] Inference refers to data processing using a trained AI model (a trained model can be called an inference model). Actual data is fed into the inference model for processing, resulting in the corresponding inference results. Inference can also be called prediction or decision-making, and the inference result can also be called a prediction result or a decision result.

[0099] 4. Neural network (NN)

[0100] Neural networks are a specific implementation of machine learning in AI. They are network structures that process information by mimicking the behavioral characteristics of animal neural networks. A neural network's structure consists of a large number of interconnected nodes (called neurons). Based on a specific computational model, neural networks process information by learning and training input. According to the universal approximation theorem, neural networks can theoretically approximate any continuous function, enabling them to learn arbitrary mappings. Therefore, neural networks can accurately abstractly model complex, high-dimensional problems. In other words, AI models can be implemented using neural networks.

[0101] A neural network is generally a multi-layer structure, consisting of multiple layers, each of which may include one or more neurons. A layer can be called a neuron layer or a network layer. Among them, the depth of a neural network refers to the number of layers contained in the neural network, and the number of neurons contained in each layer can be called the width of the layer. A neural network includes an input layer, a hidden layer, and an output layer. The input layer is responsible for receiving input signals, the output layer is responsible for outputting the calculation results of the neural network, and the hidden layer is responsible for complex functions such as feature expression. The function of the hidden layer is characterized by weights and corresponding activation functions. As shown in Figure 2, it is a schematic diagram of the fully connected layer of a neural network. The first layer in the neural network is the input layer, which includes 8 neurons. Starting from the next layer of the input layer, that is, the second layer, there are 3 hidden layers, hidden layer 1, hidden layer 2 and hidden layer 3. Each hidden layer contains 6 neurons. The number of neurons contained in different hidden layers can also be different, and this application does not limit this. The last layer of the neural network is the output layer, which contains 3 neurons. A fully connected layer means that all neurons between two adjacent layers are connected. Since a neuron in one layer is connected to every neuron in the next layer, the number of weights between two adjacent layers is equal to the product of the number of neurons in the two layers. As shown in Figure 2, the number of weights between the input layer and hidden layer 1 is 48, the number of weights between hidden layer 1 and hidden layer 2 is 36, the number of weights between hidden layer 2 and hidden layer 3 is 36, and the number of weights between hidden layer 3 and the output layer is 18.

[0102] Neural networks can also include convolutional layers, which extract important features of the data through weight sharing (or weight sharing). Convolutional layers can implement locally connected neural networks, reducing the number of weights compared to fully connected layers. The input data of a convolutional layer can include at least one channel, and a convolutional layer can include at least one convolution kernel. Each convolution kernel corresponds to a channel of the output data. The number of channels in the output data of a convolutional layer is equal to the number of convolution kernels in the convolutional layer. As shown in Figure 3, the input data of the convolutional layer includes three channels, such as channel 0, channel 1, and channel 2 of the input data, and the dimension of each channel is 5×5. The convolution kernel can include Q convolution kernels, where Q is a positive integer. The Q convolution kernels include convolution kernel q, where 1≤q≤Q. Each convolution kernel includes three channels that correspond one-to-one to the three channels of the input data, such as channel 0, channel 1, and channel 2 of convolution kernel q, and the dimension of each channel is 3×3. After the channel of the convolution kernel q is convolved with the channel of the corresponding input data to obtain the convolution result, the convolution results of each channel of the convolution kernel q are added to obtain the output data of the convolution kernel q. The output data of the convolution layer includes Q channels, and the Q channels correspond one-to-one to the Q convolution kernels. One channel among the Q channels is the output data of the convolution kernel q.

[0103] FIG4 is a schematic diagram of the training process of the intelligent model. In the example shown in FIG4, the intelligent model is a neural network. θ (·) It includes 4 convolutional layers and 4 fully connected layers. The AI ​​entity can obtain training data from the basic data set (such as a collection of channel data). The training data can include training samples (or sample data) and labels. The sample x is used as input to pass through the neural network f θ After processing (·), the inference result is output. This process is called the forward propagation of the sample. The loss function calculates the error between the inference result and the label. Based on the error obtained by the loss function, the AI ​​entity uses a backpropagation optimization algorithm (also known as a model optimization algorithm) to perform gradient backpropagation to optimize the neural network weights θ. The neural network is trained using a large amount of training data until the difference between the neural network output and the label is less than a preset value.

[0104] It should be noted that the training process shown in Figure 4 is illustrated by the training method of supervised learning as an example. Supervised learning is based on samples and labels, and the loss function is used to implement the training of the model. The training process of the intelligent model can also adopt unsupervised learning, using the algorithm to learn the intrinsic pattern of the sample, and realize the training of the intelligent model based on the sample. The training process of the intelligent model can also adopt reinforcement learning, which obtains the incentive signal of environmental feedback by interacting with the environment, thereby learning the strategy of solving the problem and realizing the optimization of the model. This application does not limit the training method of the model and the type of model.

[0105] After training, the AI ​​model can perform inference tasks. Actual data is input into the AI ​​model for processing, and the corresponding inference results are obtained.

[0106] 5. Federated Learning

[0107] Federated learning is a distributed AI model training method in which the training process is jointly executed by multiple devices based on local data, rather than by a single device. This solves the time-consuming data collection problem associated with centralized AI model training, enabling device resource sharing, increasing the diversity of training data, and improving training performance. Furthermore, since devices do not need to transmit local data, it can also reduce privacy and security concerns for some private data.

[0108] As shown in Figure 5, communication device 0, communication device 1, communication device 2, ..., communication device N can jointly perform model training. Communication device 0 can provide services for other communication devices (such as one or more of communication devices 1, 2, ..., N) to perform AI tasks, and / or communication device 0 can jointly complete AI tasks with other communication devices. However, the present application is not limited to this.

[0109] For example, at least one communication device as shown in FIG5 can perform federated learning, where communication device 0 can serve as a central device that provides model parameters (such as weights, or gradients used to determine weights) to each communication device. Communication devices 0, 1, 2, ..., and N can serve as distributed devices to obtain model parameters from the central device. After updating the model based on the model parameters, each device trains the updated model using a local dataset. For example, communication device 1 trains the model using local dataset 1, communication device 2 trains the model using local dataset 2, and communication device N trains the model using local dataset N. After model training, the multiple communication devices send the training results of this training to the central node, such as the training results including the model weights or the gradients corresponding to the weights. The central device obtains the training results from the multiple communication devices, merges the training results from the multiple communication devices, updates the model parameters based on the merged training results, and notifies each communication device. Each communication device then performs the next model training. The above process repeats until the training end condition is met, completing the model training.

[0110] The following introduces some specific application scenarios of federated learning. For example, multiple image acquisition devices use their own collected image data to perform federated learning model training on the image recognition model. After completing federated learning, the image recognition model can identify objects in the image based on the input image. For another example, in the Internet of Vehicles, the vehicle's camera, positioning system, and inertial measurement unit (IMU) collect different types of data respectively. These devices can use the collected data to perform federated learning model training on the traffic condition inference model. After completing federated learning, the vehicle can input the collected environmental data into the traffic condition inference model to obtain the traffic conditions inferred by the model to assist vehicle driving, etc.

[0111] Federated learning can significantly improve wireless communication systems' performance. In most scenarios, network devices and terminals require matching AI models to improve wireless communication performance. Federated learning can be used to train models between network devices and one or more terminals. For example, federated learning can be used to train an encoder model for channel state information (CSI) feedback compression, allowing the trained model to be used for CSI feedback compression.

[0112] Artificial intelligence applications in the communications field often involve complex nonlinear function fitting. Intelligent models are often large in scale, such as having many model layers and a large number of model weights. For example, the convolutional neural network (CNN) of the encoder model implementing CSI feedback compression may include 15 layers of neurons. The widely used residual network (ResNet) in the AI ​​field includes 34 layers of neurons. For example, the number of weights / gradients of the ResNet-20 model reaches 269,722. The visual geometry group (VGG) neural network includes 19 layers of neurons. For example, the number of weights / gradients of the VGG-16 model reaches 14,728,266. For large-scale AI models, the number of weights / gradients can reach millions or even tens of millions. When multiple communication devices jointly perform model training (such as federated learning), the model weights / gradients are transferred between the communication devices, resulting in significant data transmission overhead. Especially when applied to wireless communication networks, the occupied air interface resources may affect the normal communication services of the terminal. Therefore, in order to facilitate the application of joint model training in communication networks to improve communication performance, it is also necessary to solve the transmission overhead problem of parameters (such as weights / gradients).

[0113] This application proposes that, based on the derivation of gradients during model training, the gradient set corresponding to a layer of an intelligent model can be decomposed into a gradient accumulation factor set and an input data set. A gradient accumulation factor in the gradient accumulation factor set can be combined with the input data set to determine multiple gradients in the gradient set. When multiple communication devices jointly perform model training, the communication devices transmit the gradient accumulation factor set and the input data set instead of the gradient set, thereby reducing transmission overhead and improving the utilization of communication resources.

[0114] 6 is a schematic flow chart of a model training method 600 provided in an embodiment of the present application. In the method 600, the first communication device and the second communication device can jointly perform training of the intelligent model.

[0115] S601, the first communication device executes the model training process of the intelligent model to determine the input data set of the lth layer of the intelligent model and the gradient accumulation factor set corresponding to the lth layer.

[0116] The intelligent model can be called an AI model, an ML model, an AI / ML model or a neural network. This application does not limit the specific name of the intelligent model.

[0117] The model training process performed by the first communication device on the intelligent model includes forward propagation of training samples (or sample data) and backpropagation of gradients. The forward propagation of training samples involves inputting the training samples as input data into the input layer of the intelligent model, passing them through each layer of the intelligent model, and obtaining output data from the output layer, which is the output result of the current training of the intelligent model. The first communication device calculates the error between the output result and the label using a loss function, and then determines the gradient corresponding to each layer through backpropagation of gradients.

[0118] In one embodiment, the first layer of the intelligent model is a fully connected layer, the first layer includes N nodes, the l+1 layer includes M nodes, and N and M are positive integers greater than 1. Then the output a of the forward propagation of the first layer is l It can be expressed as follows: l =σ(z l )=σ(W l a l-1 +b l ) (1)

[0119] Among them, a l The dimension is M×1, that is, a l It includes M data corresponding to the M nodes of the l+1th layer, which are the M output data of the lth layer and the M input data of the l+1th layer. l is the output data set of the lth layer after the activation function σ, z lis the output data set of the lth layer without the activation function σ, a l-1 is the output data set of the l-1th layer after the activation function σ, that is, the input data set of the lth layer, a l-1 The dimension of W is N×1, including N data corresponding to the N nodes in the lth layer. l is the weight set between the lth layer and the l+1th layer, W l The dimension is M×N, b l is the bias of the weighted sum of input data, b l The dimension is M×1.

[0120] When the gradient is back-propagated to the lth layer, the gradient corresponding to the lth layer can be expressed as follows: g l =δ l (σ(z l-1 )) T =δ l (a l-1 ) T (2)

[0121] Among them, g l is the gradient set corresponding to the lth layer, and one gradient in the gradient set corresponds to the weight set W of the lth layer l A weight in the gradient can be used to determine the weight corresponding to the gradient, g l The dimension of is M×N. l is the set of gradient accumulation factors corresponding to the lth layer, is the loss function L on z l The partial derivative is obtained, δ l The dimension is M×1. Among them, (x) T represents the transpose of x. Specifically, the above formula (2) can be expressed as follows:

[0122] If the first communication device transmits the gradient obtained by model training to the second communication device after performing model training, then for the lth layer, the number of gradients that the first communication device needs to transmit is T = M × N. From the above formula (2-1), it can be seen that the gradient set of each layer can be decomposed into the gradient accumulation factor set and the input data set corresponding to the layer, wherein each gradient accumulation factor in the gradient accumulation factor set is combined with the input data set to determine the N gradients in the gradient set. The gradient is the gradient accumulation factor corresponding to the mth node in the l+1th layer Input data corresponding to the nth node of the lth layer The product of , 1≤m≤M, 1≤n≤N.

[0123] If the first communication device transmits the gradient accumulation factor set and input data set corresponding to layer 1 instead of the gradient set, the number of parameters the first communication device needs to transmit is M+N. When M and N are greater than 2, M+N is less than M×N, which can reduce the transmission overhead of the model parameters. The first communication device can use this method to transmit model parameters for each layer of the intelligent model, especially when each layer contains a large number of nodes and the number of training times is high, which can further reduce transmission overhead.

[0124] In another embodiment, the first layer of the intelligent model is a convolutional layer, the input data set of the first layer includes input data of P channels, the first layer includes Q convolution kernels, where P and Q are positive integers, and the output a of the forward propagation of the first layer is l It can be expressed as follows: l =σ(z l )=σ(conv(W l ,a l-1 )+b l ) (3)

[0125] Among them, conv(x,y) represents the convolution of x and y, a l is the output data set of the lth layer after the activation function σ, z l is the output data set of the lth layer without the activation function σ, a l-1 It is the output data of the l-1 layer after the activation function, and it is also the input data of the l layer. l-1 Including P channels of input data, if the dimension of the input data of each channel is Then a l-1 The dimension can be written as W l Including the weights in Q convolution kernels, each of the Q convolution kernels includes the weights of P channels, and the weights of the P channels correspond one-to-one to the P channels of the input data set. If the dimension of a channel of the convolution kernel is Then W l The dimension can be written as The weights of the P channels of each convolution kernel in the Q convolution kernels are convolved with the input data of the corresponding P channels and summed to obtain the output data of one channel corresponding to each convolution kernel, and the output data of Q channels corresponding to the Q convolution kernels. Therefore, the output data set a of the lth layer is l Includes Q channels of output data. If the dimension of the output data of each channel is Then a l The dimension can be written as The specific processing flow of the convolutional layer can be referred to the introduction of the example shown in Figure 3 above, which will not be repeated here.

[0126] The dimension of the output data of a channel The dimension of the input data The amount of padding when performing convolution and The dimension of the convolution kernel channel The convolution step size s is related to the amount of padding data. and The dimension of the input data of a channel of the convolution layer is made the same as the dimension of the output data of a channel, such as the parameters satisfy the following equations (4) and (5).

[0127] in, Indicates rounding x down.

[0128] When the gradient is back-propagated to the lth layer, the gradient corresponding to the lth layer can be expressed as follows: g l =conv(δ l ,(σ(z l-1 )) T )=conv(δ l ,(a l-1 ) T ) (6)

[0129] Among them, g l is the gradient set corresponding to the lth layer, and one gradient in the gradient set corresponds to the weight set W of the lth layer l A weight, g l The dimension is δ l is the set of gradient accumulation factors corresponding to the lth layer, is the loss function L on z l The partial derivative is obtained, δ l The dimension is

[0130] The gradient accumulation factor set δ corresponding to layer l l It includes Q gradient accumulation factor subsets corresponding to Q convolution kernels. Each gradient accumulation factor subset combined with the input data set can determine the gradient corresponding to the P channels of the convolution kernel corresponding to the gradient accumulation factor subset. For example, the gradient accumulation factor subset corresponding to the convolution kernel q With the input data set a l-1 , The dimension is The gradient corresponding to the P channels corresponding to the convolution kernel q can be determined The dimension is

[0131] Therefore, after the first communication device performs model training, it can transmit the gradient accumulation factor set δ obtained by the gradient decomposition corresponding to the lth layer to the second communication device. l and input data set a l-1 Instead of transmitting the gradient set g l , which can reduce the transmission overhead of model parameters. The first communication device can use this method to transmit model parameters for each layer of the intelligent model. Especially when the dimension of the input data is small and the number of training times is large, the transmission overhead can be reduced to a greater extent.

[0132] S602: The first communication device sends first information to the second communication device, where the first information is used to indicate an input data set of the lth layer and a gradient accumulation factor set corresponding to the lth layer.

[0133] Correspondingly, the second communication device receives the first information from the first communication device, and determines the input data set of the lth layer and the gradient accumulation factor set corresponding to the lth layer based on the first information, so that the second communication device can determine the gradient set corresponding to the lth layer obtained by the first communication device after this training of the intelligent model based on the input data set of the lth layer and the gradient accumulation factor set corresponding to the lth layer in S603.

[0134] Specifically, the first information can indicate the input data set of each layer of the intelligent model and the gradient accumulation factor set corresponding to each layer, so that the second communication device can determine the gradient set corresponding to each layer obtained by the first communication device after training the intelligent model this time based on the input data set of each layer and the gradient accumulation factor set corresponding to each layer in S603.

[0135] S603: The second communication device determines a gradient set corresponding to the lth layer according to the input data set of the lth layer and the gradient accumulation factor set corresponding to the lth layer.

[0136] For example, the second communication device may be based on the above formula (2) according to the input data set a of the first layer. l-1 and the gradient accumulation factor set δ corresponding to the lth layer l , obtain the gradient set g corresponding to the lth layer obtained by the first communication device after this model training l .

[0137] Optionally, the second communication device may be configured to generate a gradient set g corresponding to the first layer. l , determine the weight set W of the lth layer l The second communication device may determine an updated intelligent model based on the weight set.

[0138] Specifically, the second communication device may obtain, from the first information, the input data set corresponding to each layer of the intelligent model and the gradient accumulation factor set corresponding to each layer. The second communication device may then determine the gradient set corresponding to each layer based on the first information. Alternatively, the second communication device may determine the weight set for each layer based on the gradient set corresponding to each layer, thereby determining an updated intelligent model based on the weight set for each layer.

[0139] According to the above-mentioned solution of the application, when multiple communication devices jointly perform model training and need to transmit the gradient set or weight set of a neuron layer between communication devices, the input data set and the corresponding gradient accumulation factor set obtained by decomposing the gradient set corresponding to the neuron layer can be transmitted. The receiving device can determine the gradient set or weight set based on the input data set and the gradient accumulation factor set. This model parameter transmission method can reduce transmission overhead and improve resource utilization.

[0140] As shown in Table 1, taking the intelligent model as a neural network with fully connected layers as an example, the five layers of Neural Network 1 contain 2, 25, 50, 25, and 2 neurons, respectively. After a single model training run, if the first communication device transmits the gradient set corresponding to each layer, the number of gradients (i.e., the number of parameters) that need to be transmitted is 2600 (i.e., 2×25+25×50+50×25+25×2). If the gradients are decomposed and the gradient accumulation factor set and input data set corresponding to each layer are transmitted, the number of model parameters (including the gradient accumulation factor and input data) that need to be transmitted is 204 (i.e., 2+25+25+50+50+25+25+2). This shows that the transmission overhead after a single training run is reduced to less than one-tenth of the original. For neural networks with a larger number of neurons per layer, as shown in Table 1, the five layers of Neural Network 2 contain 16, 512, 1024, 1024, and 16 neurons, respectively. If the first communication device transmits the gradient set corresponding to each layer obtained by the training after performing one model training, the number of gradients that need to be transmitted (i.e., the number of parameters) is 1597440. If the gradient is decomposed and the gradient accumulation factor set and input data set corresponding to each layer are transmitted, the number of model parameters (including gradient accumulation factors and input data) that need to be transmitted is 5184. It can be seen that through the scheme provided in this application, the transmission overhead can be greatly reduced after one model training, and the transmission overhead can be further reduced for multiple training of the model, thereby improving resource utilization.

[0141] Table 1

[0142] As previously described, a first communication device can indicate, via first information, the input data set of layer 1 of an intelligent model and the set of gradient accumulation factors corresponding to layer 1, so that a second communication device can determine, based on the first information, the gradient set or weight set corresponding to layer 1 of the intelligent model. In one embodiment, the first communication device can indicate, via the first information, the input data set of each layer of the intelligent model and the set of gradient accumulation factors corresponding to each layer, so that the second communication device can determine, based on the first information, the gradient set corresponding to each layer of the intelligent model.

[0143] The present application also provides another embodiment, in which the first information may indicate an input data set of each layer of the intelligent model and a gradient accumulation factor component set corresponding to each layer. A gradient accumulation factor component in the gradient accumulation factor component set is a component of a gradient accumulation factor in the gradient accumulation factor set, and the gradient accumulation factor component set has the same dimension as the gradient accumulation factor set.

[0144] If the first layer of the intelligent model is a fully connected layer, the first communication device can be based on the gradient factor set δ corresponding to the l+1 layer l+1 , weight set W l+1 And the input data set a of the l+1 layer l (i.e. the output data set a of the lth layer l ) Using partial derivative operations and / or multiplication operations, determine the set of gradient accumulation factors corresponding to the lth layer, for example, it can be expressed as follows: δ l =(W l+1 ) T δ l+1 ⊙σ'(z l ) (7)

[0145] Among them, σ'(z l ) represents σ(z l ), that is, the output data set a of the lth layer l The derivative of , ⊙ represents the Hadamard dot product. Among them, (W l+1 ) T δ l+1 is the gradient accumulation factor set δ l The gradient accumulation factor component set, (W l+1 ) T δ l+1 The dimension is M×1.

[0146] For example, the intelligent model is shown in FIG6A , with the activation function For example, the derivative of the activation function σ'(x) can be expressed as follows:

[0147] The gradient set of layer l can be expressed as

[0148] In the above formula (7), W l+1 The dimension is H×M, H is the number of nodes contained in the l+2 layer, δ l+1 The dimension is H×1, we can get:

[0149] Among them, σ(z l )=a l , therefore,

[0150] Here, m is an integer greater than or equal to 1 and less than or equal to M.

[0151] If l is equal to L, that is, if the lth layer is the output layer of the intelligent model, the first communication device determines the gradient accumulation factor set corresponding to the Lth layer based on the output data set and label set of the Lth layer, and the label set is used to determine the weight set of the intelligent model.

[0152] In the example shown in FIG6A , the gradient set of the Lth layer can be expressed as:

[0153] J is the number of nodes contained in the L-1 layer. L =(aL-t)⊙σ'(z l )

[0154] Among them, t is the label set, and the dimension is K×1. Then therefore,

[0155] Here, k is an integer greater than or equal to 1 and less than or equal to K.

[0156] If the first layer of the intelligent model is a convolutional layer, the first communication device can be based on the gradient factor set δ corresponding to the l+1 layer l+1 , weight set W l+1 And the input data set a of the l+1 layer l (i.e. the output data set a of the lth layer l ) Using partial derivative operations and / or convolution operations, determine the set of gradient accumulation factors corresponding to the lth layer, for example, it can be expressed as follows: δ l =conv(rot180(W l+1 ) T ,δ l+1 )⊙σ'(z l ) (8)

[0157] Among them, conv(rot180(W l+1 ) T ,δl+1 ) is the gradient accumulation factor set δ l The gradient accumulation factor component set is the same as the gradient accumulation factor. l+1 ) T Represents the weight W l+1 Rotate 180 degrees. Rotating 180 degrees means sorting the rows of a matrix in reverse order, and the elements of each row are also sorted in reverse order. For example, for a matrix W with dimensions H×M l+1 :

[0158] To W l+1 Rot180(W l+1 )for:

[0159] It can be seen from the above formulas (7) and (8) that the gradient accumulation factor set corresponding to the lth layer can be obtained by the gradient accumulation factor component set and the input data set of the l+1th layer. Therefore, the first communication device can send a first message to the second communication device, and the first message is used to indicate the gradient accumulation factor component set corresponding to each layer of the intelligent model and the input data set of each layer obtained by the first communication device performing model training. After receiving the first message, the second communication device can determine the gradient accumulation factor set corresponding to each layer based on the gradient accumulation factor component set of each layer and the output data set of each layer (i.e., the input data set of the next layer) according to the above formula (7) or (8), and then determine the gradient set corresponding to each layer according to formula (2) or (6). Optionally, the weight set corresponding to each layer can also be determined based on the gradient set corresponding to each layer, thereby determining the updated intelligent model.

[0160] In this embodiment, the indication overhead of the first information is the same as the overhead of the input data of each layer and the corresponding gradient accumulation factor set indicated by the first information. Compared with the feedback gradient set, it can reduce transmission overhead and improve resource utilization.

[0161] Figure 7 is a schematic flow chart of a communication method 700 provided in an embodiment of the present application. In this method 700, communication device 0 to communication device 2 may correspond to communication device 0 to communication device 2 shown in Figure 5, and the communication device 0 and communication device 2 perform model training in a federated learning manner. Communication device 0, as a central device, provides model parameters to communication device 1 and communication device 2, which are distributed devices. Communication device 1 and communication device 2 respectively perform model training based on local data and then feedback the training results to communication device 0 using the method shown in Figure 6. After communication device 0 integrates the training results of communication device 1 and communication device 2, it provides updated model parameters to communication device 1 and communication device 2. Communication device 1 and communication device 2 determine the updated model based on the updated model parameter set and perform the next model training. The method may also include other communication devices. The method 700 is illustrated by taking the three communication devices as an example. The method includes but is not limited to the following steps:

[0162] S701, the communication device 1 performs the nth model training process of the intelligent model to determine the input data set 1 and the gradient accumulation factor set 1.

[0163] The communication device 1 may receive a model parameter set corresponding to the n-1th model training from the communication device 0, and determine an updated intelligent model based on the model parameter set corresponding to the n-1th model training. For example, the model parameter set may be obtained by the communication device 0 by fusing the results of the n-1th model training of the communication device 0 and the communication device 1. If n=1, the indication information may indicate the initial model parameter set of the intelligent model. The model parameter set may be a gradient set or a weight set of the intelligent model.

[0164] After determining an updated intelligent model based on the model parameter set corresponding to the n-1th model training, communication device 1 uses the updated intelligent model to perform the nth model training process, obtaining input data set 1 and gradient accumulation factor set 1. Input data set 1 includes the input data set corresponding to each layer of the intelligent model obtained in this model training, and gradient accumulation factor set 1 includes the gradient accumulation factor set corresponding to each layer of the intelligent model obtained in this model training. The specific determination method can be referred to the description of the embodiment shown in FIG. 6 above and will not be repeated here.

[0165] S702, the communication device 2 performs the nth model training process of the intelligent model to determine the input data set 2 and the gradient accumulation factor set 2.

[0166] The communication device 2 determines the input data set 2 and the gradient accumulation factor 2 in the same manner as the communication device 1 in S701 .

[0167] It should be understood that this application does not limit the order in which each communication device executes the steps in the embodiment shown in FIG. 7 . The order of the steps is determined by the logical relationship between the steps. In specific implementations, the order of the steps can be changed if there is no logical conflict. For example, S702 can be performed before S701, and the subsequent S705 can be performed before S703, or S704 can be performed after S705.

[0168] S703, communication device 1 sends information 1 to communication device 0, where the information 1 is used to indicate the input data set 1 and gradient accumulation factor set 1 of the intelligent model.

[0169] Accordingly, the communication device 0 receives the information 1 from the communication device 1 .

[0170] S704 , the communication device 0 determines a gradient set 1 according to the input data set 1 and the gradient accumulation factor set 1 .

[0171] The communication device 0 can determine the gradient set 1 based on the type of each layer of the intelligent model (such as a fully connected layer or a convolutional layer) based on formula (2) or formula (6), and the gradient set 1 includes the gradient sets corresponding to each layer determined by the communication device 1 performing the nth model training.

[0172] S705 , the communication device 2 sends information 2 to the communication device 0 , where the information 2 is used to indicate the input data set 2 and the gradient accumulation factor set 2 of the intelligent model.

[0173] Accordingly, the communication device 0 receives the information 2 from the communication device 2 .

[0174] S706 , the communication device 0 determines a gradient set 2 according to the input data set 2 and the gradient accumulation factor set 2 .

[0175] Similarly, S705 and S706 can be implemented with reference to S703 and S704.

[0176] S707 , the communication device 0 determines a model parameter set corresponding to the n-th model training according to the gradient set 1 and the gradient set 2 .

[0177] After communication device 0 obtains gradient set 1 and gradient set 2 determined by communication device 1 and communication device 2 performing the nth model training based on information 1 and information 2 respectively, communication device 0 fuses the training results of the two devices to obtain the model parameter set corresponding to the nth model training.

[0178] S708, communication device 0 sends indication information to communication device 1 and communication device 2, where the indication information is used to indicate a model parameter set corresponding to the nth model training.

[0179] Accordingly, communication devices 1 and 2 receive the instruction from communication device 0 and, based on the model parameter sets corresponding to the nth round of model training, determine updated intelligent models. They then perform model training for the n+1th round based on the updated intelligent models. Once communication device 0 determines that the conditions for model training have been met, it notifies communication devices 1 and 2 to stop model training. Each communication device then obtains a trained intelligent model, which can be used for inference.

[0180] Exemplarily, multiple communication devices use federated learning to train intelligent models. These devices may include access network nodes and terminals, or they may be communication devices in the Internet of Things, vehicular networks, or industrial Internet of Things. Federated learning can improve the diversity of model training data and the sharing of device processing capabilities.

[0181] After model training, the intelligent model can be used to perform physical layer reasoning tasks, such as the intelligent model being used to reason about compressed CSI based on CSI. After the access network node and multiple terminals can jointly train the intelligent model, the multiple terminals can perform CSI compression based on the intelligent model, thereby feeding back the compressed CSI to the access network node. Alternatively, after model training, the intelligent model can be used to perform application layer reasoning tasks, such as the intelligent model being used for speech recognition. If the application layers of multiple terminals need to perform speech recognition tasks, the access network node can configure multiple terminals to jointly train the intelligent model. After training is completed, the multiple terminals respectively use the intelligent model to perform speech recognition reasoning tasks.

[0182] According to the solution provided by this application, when multiple communication devices jointly perform model training and need to transmit the gradient set or weight set of a neuron layer between communication devices, the input data set and the corresponding gradient accumulation factor set obtained by decomposing the gradient set corresponding to the neuron layer can be transmitted. The receiving device can then determine the gradient set or weight set based on the input data set and the gradient accumulation factor set. This method of transmitting model parameters can reduce transmission overhead and improve resource utilization.

[0183] It is understood that, in order to implement the functions in the above embodiments, the base station and the terminal include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily appreciate that, in conjunction with the units and method steps of the various examples described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a computer software-driven hardware manner depends on the specific application scenario and design constraints of the technical solution.

[0184] Figures 8 to 10 are schematic diagrams of the structures of possible communication devices provided in embodiments of the present application. These communication devices can be used to implement the functions of the communication devices in the above-mentioned method embodiments, and thus can also achieve the beneficial effects possessed by the above-mentioned method embodiments. In the embodiments of the present application, when the communication device is used to implement the functions of the communication device in the above-mentioned method embodiments, the communication device can be one of the terminals 120a-120j shown in Figure 1. Alternatively, the communication device can be the RAN node 110a or 110b shown in Figure 1, or it can also be a module (such as a chip or chip system) applied to a terminal or network device.

[0185] The communication device 800 shown in FIG8 includes a transceiver unit 820 , which can be used to receive or send information. The communication device 800 may also include a processing unit 810 , which can be used to process instructions or data to implement corresponding operations.

[0186] It should be understood that when the communication device 800 is a chip configured in (or used in) a communication device, the transceiver unit 820 in the communication device 800 can be the input / output interface or circuit of the chip, and the processing unit 810 in the communication device 800 can be the processor in the chip.

[0187] Optionally, the communication device 800 may further include a storage unit, which may be used to store instructions or data. The processing unit 810 may execute the instructions or data stored in the storage unit to enable the communication device to perform corresponding operations.

[0188] Communication device 800 can be used to implement the functions of the second communication device or communication device 0 in the above-mentioned method embodiment. When communication device 800 is used to implement the functions of the second communication device or communication device 0 in the above-mentioned method embodiment: transceiver unit 820 is used to receive first information, wherein the first information is used to indicate the input data set of layer 1 of the intelligent model and the gradient accumulation factor set corresponding to the layer 1, where 1 is a positive integer. Processing unit 810 is used to determine the gradient set corresponding to the layer 1 based on the input data set of the layer 1 and the gradient accumulation factor set corresponding to the layer 1, wherein a gradient accumulation factor in the gradient accumulation factor set is used to determine multiple gradients in the gradient set, and the gradient set is used to determine the intelligent model.

[0189] Communication device 800 can be used to implement the functions of the first communication device, communication device 1, or communication device 2 in the above-mentioned method embodiments. When communication device 800 is used to implement the functions of the first communication device, communication device 1, or communication device 2 in the above-mentioned method embodiments: Processing unit 810 is used to execute a model training process for an intelligent model, determine an input data set for layer 1 of the intelligent model, and a set of gradient accumulation factors corresponding to layer 1, where 1 is a positive integer. Transceiver unit 820 is used to send first information indicating the input data set for layer 1 and the set of gradient accumulation factors corresponding to layer 1, wherein a gradient accumulation factor in the set of gradient accumulation factors is used to determine multiple gradients in the set of gradients corresponding to layer 1.

[0190] For a more detailed description of the processing unit 810 and the transceiver unit 820 , reference may be made to the relevant description in the above method embodiment.

[0191] It should be understood that the transceiver unit 820 in the communication device 800 can be implemented by, for example, a transceiver, a transceiver circuit, an input / output interface, or a pin. When the transceiver unit 820 is a transceiver, the transceiver can be composed of a receiver and / or a transmitter. The processing unit 810 in the communication device 800 can be implemented by at least one processor, or by at least one logic circuit. Optionally, the communication device 800 also includes a storage unit, which can be implemented by a memory.

[0192] As shown in Figure 9, communication device 900 includes a processor 910 and an interface circuit 920. Processor 910 and interface circuit 920 are coupled to each other. It is understood that interface circuit 920 can be a transceiver or an input / output interface. Optionally, communication device 900 may also include a memory 930 for storing instructions executed by processor 910, input data required by processor 910 to execute instructions, or data generated after processor 910 executes instructions.

[0193] In one implementation, the memory 930 may also be integrated into the processor 910 or independent of the processor 910 .

[0194] When the communication device 900 is used to implement the method provided by the above method embodiment, the processor 910 is used to implement the functions of the above processing unit 810, and the interface circuit 920 is used to implement the functions of the above transceiver unit 820.

[0195] Figure 10 shows another schematic diagram of the structure of a communication device 1100. It is understood that the communication device 1100 includes devices in necessary forms such as modules, units, elements, circuits, or interfaces, which are appropriately configured together to implement the functions of the first communication device or the second communication device in the above-mentioned embodiments of the present application. The communication device 1100 can be a RAN node, terminal, core network device, or other network device in Figure 1, or a component (such as a chip) in these devices, used to implement the method described in the above-mentioned method embodiment. The communication device 1100 includes one or more processors 1010. The processor 1010 can be a general-purpose processor or a dedicated processor. For example, it can be a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, and the central processing unit can be used to control the communication device (such as a RAN node, terminal, or chip), execute software programs, and process software program data.

[0196] Optionally, in one design, the processor 1010 may include a program 1011 (sometimes also referred to as code or instructions), which may be executed on the processor 1010 to enable the communication device 1100 to perform the methods described in the above embodiments. In another possible design, the communication device 1100 may include a circuit (not shown in FIG. 10 ) configured to implement the functions of the communication device in the above embodiments of receiving and / or sending signals.

[0197] Optionally, the communication device 1100 may include one or more memories 1020 on which a program 1021 (sometimes also referred to as code or instructions) is stored. The program 1021 can be run on the processor 1010, so that the communication device 1100 executes the method described in the above method embodiment.

[0198] Optionally, the processor 1010 and / or the memory 1020 may include an AI module 1012, 1022, which is configured to implement AI-related functions. The AI ​​module may be implemented using software, hardware, or a combination of software and hardware. For example, the AI ​​module may include a real-time information processing (RIC) module. For example, the AI ​​module may be a near-real-time RIC or a non-real-time RIC.

[0199] Optionally, data may be stored in the processor 1010 and / or the memory 1020. The processor and the memory may be provided separately or integrated together.

[0200] Optionally, the communication device 1100 may further include a transceiver 1030 and / or an antenna 1040. The processor 1010 may also be sometimes referred to as a processing unit, and controls the communication device (e.g., a RAN node or terminal). The transceiver 1030 may also be sometimes referred to as a transceiver unit, a transceiver, a transceiver circuit, or a transceiver, and is configured to implement the transceiver functions of the communication device through the antenna 1040.

[0201] When the communication device is a chip used in a terminal, the chip can implement the terminal-related functions in the above method embodiments. The chip receives information / data from other modules in the terminal (such as a radio frequency module or antenna), and the information / data may be sent by a network device to the terminal; or the chip sends information / data to other modules in the terminal (such as a radio frequency module or antenna), and the information / data may be sent by the terminal device to the network device.

[0202] When the above-mentioned communication device is a module applied to a network device, the module can implement the functions related to the network device in the above-mentioned method embodiment. The module receives information / data from other modules in the network device (such as a radio frequency module or an antenna), and the information / data can be sent by the terminal to the network device; or, the module sends information / data to other modules in the network device (such as a radio frequency module or an antenna), and the information / data can be sent by the network device to the terminal. The network device module here can be a chip of the network device, or it can be a DU or other module (for example, a CU or a radio unit (RU), etc.), wherein the RU can include a remote radio unit (RRU). The DU can be an O-DU under the O-RAN architecture, the CU can be an O-CU under the O-RAN architecture, and the RU can be an O-RU under the O-RAN architecture.

[0203] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0204] The method steps in the embodiments of the present application can be implemented in hardware or in software instructions that can be executed by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disk, mobile hard disk, CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. The storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in an access network device or a terminal device. The processor and storage medium can also exist in the access network device or the terminal device as discrete components.

[0205] According to the method provided in the embodiment of the application, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is executed by one or more processors, it enables the device including the processor to execute the method provided in the above method embodiment.

[0206] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are performed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device.

[0207] According to the method provided in the embodiments of the present application, the embodiments of the present application also provide a computer-readable storage medium, which stores the above-mentioned computer program or instructions. When the computer program or instructions are executed by one or more processors, the device including the processor executes the method provided in the above-mentioned method embodiment.

[0208] As described above, the computer program or instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video disk; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0209] According to the method provided in the embodiment of the present application, the embodiment of the present application also provides a communication system, including the one or more terminals mentioned above. The system may further include the one or more network devices mentioned above.

[0210] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the devices described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0211] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this solution based on actual needs.

[0212] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0213] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A model training method, characterized in that: include: Receive first information, where the first information is used to indicate an input data set of a lth layer of the intelligent model and a gradient accumulation factor set corresponding to the lth layer, where l is a positive integer; According to the input data set of the lth layer and the gradient accumulation factor set corresponding to the lth layer, the gradient set corresponding to the lth layer is determined, wherein a gradient accumulation factor in the gradient accumulation factor set is used to determine multiple gradients in the gradient set, and the gradient set is used to determine the intelligent model.

2. The method according to claim 1, characterized in that The method further comprises: Determine a weight set of the intelligent model according to the gradient set, wherein the gradient set includes a gradient corresponding to each weight in the weight set; The updated intelligent model is determined according to the weight set.

3. The method according to claim 1 or 2, characterized in that: The first layer is a fully connected layer, the first layer includes N nodes, the l+1th layer of the intelligent model includes M nodes, and N and M are positive integers greater than 2; The input data set of the lth layer includes N input data corresponding to the N nodes, and the gradient accumulation factor set corresponding to the lth layer includes M gradient accumulation factors corresponding to the M nodes, wherein the gradient set includes T gradients, and T is the product of N and M.

4. The method according to claim 3, characterized in that: Determining the gradient set corresponding to the lth layer according to the input data set of the lth layer and the gradient accumulation factor set corresponding to the lth layer includes: According to the input data of each node in the N nodes and the gradient accumulation factor corresponding to each node in the M nodes, T gradients in the gradient set are determined, and a gradient g in the T gradients is mn It is the product of the input data of the nth node among the N nodes and the gradient accumulation factor corresponding to the mth node among the M nodes, n is a positive integer less than or equal to N, and m is a positive integer less than or equal to M.

5. The method according to claim 1 or 2, characterized in that: The input data set of the lth layer includes input data of P channels; The first layer includes Q convolution kernels, the Q convolution kernels correspond to Q channels of output data of the first layer, and the set of gradient accumulation factors corresponding to the first layer includes Q gradient accumulation factor subsets corresponding to the Q convolution kernels; The gradient set includes R gradient subsets, the R gradient subsets include P gradient subsets corresponding to each of the Q convolution kernels, the P gradient subsets corresponding to each convolution kernel correspond to the P channels, wherein P and Q are positive integers, and R is the product of Q and P.

6. The method according to claim 5, characterized in that The step of determining a gradient set according to the input data set of the first layer and the gradient accumulation factor set corresponding to the first layer includes: According to the input data of each channel in the P channels and the gradient accumulation factor subset corresponding to each convolution kernel in the Q convolution kernels, a convolution operation is used to determine R gradient subsets in the gradient set, and a gradient subset G in the R gradient subsets is qp It is obtained by using a convolution operation based on the gradient accumulation factor corresponding to the p-th channel among the P channels and the q-th convolution kernel among the Q convolution kernels.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The gradient set is sent, and the gradient set is used to determine the intelligent model.

8. A model training method, characterized in that: include: Execute a model training process of the intelligent model to determine an input data set of the lth layer of the intelligent model and a gradient accumulation factor set corresponding to the lth layer, where l is a positive integer; Sending first information, where the first information is used to indicate an input data set of the lth layer and a gradient accumulation factor set corresponding to the lth layer, wherein a gradient accumulation factor in the gradient accumulation factor set is used to determine multiple gradients in a gradient set corresponding to the lth layer.

9. The method according to claim 8, characterized in that The step of determining a set of gradient accumulation factors corresponding to the first layer of the intelligent model includes: The gradient accumulation factor set corresponding to the lth layer is determined according to the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer and the output data set of the lth layer.

10. The method according to claim 9, characterized in that The intelligent model includes L layers, where l is less than or equal to L, and determining a set of gradient accumulation factors corresponding to the lth layer of the intelligent model includes: If l is equal to L, the gradient accumulation factor set corresponding to the Lth layer is determined according to the output data set and label set of the Lth layer, and the label set is used to determine the weight set of the intelligent model.

11. The method according to claim 9 or 10, characterized in that: The first layer is a fully connected layer, the first layer includes N nodes, the l+1th layer of the intelligent model includes M nodes, and N and M are positive integers greater than 2; The input data set of the lth layer includes N input data corresponding to the N nodes, and the gradient accumulation factor set corresponding to the lth layer includes M gradient accumulation factors corresponding to the M nodes, wherein the gradient set includes T gradients, and T is the product of N and M.

12. The method according to claim 11, characterized in that The step of determining the gradient accumulation factor set corresponding to the lth layer according to the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer comprises: According to the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer and the output data set of the lth layer, the gradient accumulation factor set corresponding to the lth layer is determined by using partial derivative operations and / or multiplication operations.

13. The method according to claim 9 or 10, characterized in that: The input data set of the lth layer includes input data of P channels; The first layer includes Q convolution kernels, the Q convolution kernels correspond to Q channels of output data of the first layer, and the set of gradient accumulation factors corresponding to the first layer includes Q gradient accumulation factor subsets corresponding to the Q convolution kernels; The gradient set includes R gradient subsets, the R gradient subsets include P gradient subsets corresponding to each of the Q convolution kernels, the P gradient subsets corresponding to each convolution kernel correspond to the P channels, and R is the product of Q and P.

14. The method according to claim 13, characterized in that The step of determining the gradient accumulation factor set corresponding to the lth layer according to the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer, and the output data set of the lth layer comprises: According to the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer and the output data set of the lth layer, the gradient accumulation factor set corresponding to the lth layer is determined by using partial derivative operations and / or convolution operations.

15. A model training device, characterized in that: include: A transceiver unit, used to receive first information, where the first information is used to indicate an input data set of a layer l of the intelligent model and a set of gradient accumulation factors corresponding to the layer l, where l is a positive integer; A processing unit is used to determine a gradient set corresponding to the lth layer based on an input data set of the lth layer and a gradient accumulation factor set corresponding to the lth layer, wherein a gradient accumulation factor in the gradient accumulation factor set is used to determine multiple gradients in the gradient set, and the gradient set is used to determine the intelligent model.

16. The device according to claim 15, characterized in that The processing unit is also used for: Determine a weight set of the intelligent model according to the gradient set, wherein the gradient set includes a gradient corresponding to each weight in the weight set; The updated intelligent model is determined according to the weight set.

17. The device according to claim 15 or 16, characterized in that The first layer is a fully connected layer, the first layer includes N nodes, the l+1th layer of the intelligent model includes M nodes, and N and M are positive integers greater than 2; The input data set of the lth layer includes N input data corresponding to the N nodes, and the gradient accumulation factor set corresponding to the lth layer includes M gradient accumulation factors corresponding to the M nodes, wherein the gradient set includes T gradients, and T is the product of N and M.

18. The device according to claim 17, characterized in that The processing unit is specifically configured to determine T gradients in the gradient set according to input data of each node in the N nodes and a gradient accumulation factor corresponding to each node in the M nodes, wherein a gradient g in the T gradients is mn It is the product of the input data of the nth node among the N nodes and the gradient accumulation factor corresponding to the mth node among the M nodes, n is a positive integer less than or equal to N, and m is a positive integer less than or equal to M.

19. The device according to claim 15 or 16, characterized in that The input data set of the lth layer includes input data of P channels; The first layer includes Q convolution kernels, the Q convolution kernels correspond to Q channels of output data of the first layer, and the set of gradient accumulation factors corresponding to the first layer includes Q gradient accumulation factor subsets corresponding to the Q convolution kernels; The gradient set includes R gradient subsets, the R gradient subsets include P gradient subsets corresponding to each of the Q convolution kernels, the P gradient subsets corresponding to each convolution kernel correspond to the P channels, and R is the product of Q and P.

20. The device according to claim 19, characterized in that The processing unit is specifically used to determine R gradient subsets in the gradient set by using a convolution operation according to the input data of each channel in the P channels and the gradient accumulation factor subset corresponding to each convolution kernel in the Q convolution kernels, and a gradient subset G in the R gradient subsets qp It is obtained by using a convolution operation based on the gradient accumulation factor corresponding to the p-th channel among the P channels and the q-th convolution kernel among the Q convolution kernels.

21. The device according to any one of claims 15 to 20, characterized in that The transceiver unit is further used to send the gradient set, and the gradient set is used to determine the intelligent model.

22. A model training device, characterized in that: include: A processing unit, used to execute a model training process of the intelligent model, determine an input data set of a layer l of the intelligent model and a set of gradient accumulation factors corresponding to the layer l, where l is a positive integer; A transceiver unit is used to send first information, where the first information is used to indicate an input data set of the lth layer and a gradient accumulation factor set corresponding to the lth layer, wherein a gradient accumulation factor in the gradient accumulation factor set is used to determine multiple gradients in a gradient set corresponding to the lth layer.

23. The device according to claim 22, characterized in that The processing unit is specifically used to determine the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer and the output data set of the lth layer.

24. The device according to claim 23, characterized in that The intelligent model includes L layers, l is less than or equal to L. If l is equal to L, the processing unit is specifically used to determine the gradient accumulation factor set corresponding to the Lth layer based on the output data set and label set of the Lth layer, and the label set is used to determine the weight set of the intelligent model.

25. The device according to claim 23 or 24, characterized in that The first layer is a fully connected layer, the first layer includes N nodes, the l+1th layer of the intelligent model includes M nodes, and N and M are positive integers greater than 2; The input data set of the lth layer includes N input data corresponding to the N nodes, and the gradient accumulation factor set corresponding to the lth layer includes M gradient accumulation factors corresponding to the M nodes, wherein the gradient set includes T gradients, and T is the product of N and M.

26. The device according to claim 25, characterized in that The processing unit is specifically used to determine the gradient accumulation factor set corresponding to the lth layer by using partial derivative operations and / or multiplication operations based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer and the output data set of the lth layer.

27. The device according to claim 23 or 24, characterized in that The input data set of the lth layer includes input data of P channels; The first layer includes Q convolution kernels, the Q convolution kernels correspond to Q channels of output data of the first layer, and the set of gradient accumulation factors corresponding to the first layer includes Q gradient accumulation factor subsets corresponding to the Q convolution kernels; The gradient set includes R gradient subsets, the R gradient subsets include P gradient subsets corresponding to each of the Q convolution kernels, the P gradient subsets corresponding to each convolution kernel correspond to the P channels, and R is the product of Q and P.

28. The device according to claim 27, characterized in that The processing unit is specifically used to determine the gradient accumulation factor set corresponding to the lth layer based on the weight set of the l+1th layer of the intelligent model, the gradient accumulation factor set corresponding to the l+1th layer and the output data set of the lth layer, using partial derivative operations and / or convolution operations.

29. A communication device, characterized in that: comprising at least one processor; The at least one processor is used to execute a program or instruction stored in the memory, so that the communication device implements the method according to any one of claims 1 to 7, or implements the method according to any one of claims 8 to 14.

30. The communication device according to claim 29, characterized in that The communication device includes the memory.

31. A computer-readable storage medium storing a computer program or instruction, wherein when the computer program or instruction is executed in a computer, the computer executes the method according to any one of claims 1 to 7, or executes the method according to any one of claims 8 to 14.

32. A communication system, characterized in that: Comprising a device as claimed in any one of claims 15 to 21, and / or a device as claimed in any one of claims 22 to 28.

33. A computer program product, characterized in that When the computer program product is executed in a computer, the computer is enabled to execute the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 14.

Images