Data processing method, device, system and related equipment

Abstract

The present application provides a data processing method, apparatus, system and related equipment, the method is applied to a neural network structure search NAS system, the system includes multiple processors, each processor is used to train a sub-network in the search super network, the method includes: each of the multiple processors performs current iterative training on the corresponding sub-network to obtain the to-be-updated part of the target parameter of each processor; each of the multiple processors transmits information of the to-be-updated part of the target parameter obtained between at least one other processor, so that each of the multiple processors obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors, and then when the multiple processors in the NAS system communicate with each other, only the to-be-updated part of the target parameter of each processor is transmitted, compared with directly transmitting all the data on the processor, the communication volume can be reduced and the training efficiency of the NAS system can be improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to methods, devices, systems and related equipment for data processing. Background Art

[0002] In recent years, in order to solve the problem that neural network structures are completely dependent on manual design, researchers have begun to try to gradually transform the design of neural network structures into machine automatic design, and thus neural network structure search (NAS) came into being. NAS refers to finding a better network structure in a given search space and trying to optimize the weight parameters of the network structure to obtain a trained network model.

[0003] NAS usually uses a recurrent neural network (RNN) as a controller to generate multiple sub-networks, and then trains and evaluates each sub-network. If the network performance (such as accuracy) of the sub-network is not good, the structure will be discarded to find an optimal network structure. Since the weights of each sub-network of NAS are trained independently, that is, each sub-network requires a complete training process, which consumes huge computing resources and time. Therefore, NAS often distributes the workload to multiple computing nodes to speed up training, but this also introduces huge communication overhead for exchanging model parameters or gradients in each iteration, which limits the training speed. Summary of the invention

[0004] The present application provides a technical solution for data processing.

[0005] In a first aspect, a data processing method is provided, which is applied to a neural network structure search NAS system, the system comprising multiple processors, each processor being used to train a subnetwork in a search supernetwork, the method comprising the following steps: each of the multiple processors performs current iterative training on the corresponding subnetwork to obtain a to-be-updated portion of a target parameter of each processor, each of the multiple processors transmits information on the to-be-updated portion of the target parameter obtained between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the to-be-updated portion of the target parameter corresponding to each of the multiple processors.

[0006] Optionally, the processor is a GPU.

[0007] By implementing the method described in the first aspect, during the training process of the NAS system, when multiple processors communicate with each other, only the to-be-updated part of the target parameter of each processor is transmitted. Compared with other methods in which all data on the processor is transmitted, the communication volume can be greatly reduced, thereby improving the training efficiency of the NAS system.

[0008] In a possible implementation, each processor encodes the part to be updated of the target parameter to obtain the mask of the part to be updated; the information of the part to be updated includes: the part to be updated and the mask of the part to be updated. The information of the part to be updated of the target parameter obtained by each processor among the multiple processors is transmitted between at least one other processor, so that each processor among the multiple processors obtains the part to be updated of the target parameter corresponding to each of the multiple processors, including: each processor among the multiple processors transmits the mask of the part to be updated between at least one other processor, so that each processor among the multiple processors obtains the mask of the part to be updated corresponding to each of the multiple processors; each processor among the multiple processors obtains the union of the masks of the parts to be updated corresponding to each of the multiple processors; each processor among the multiple processors determines the overall part to be updated corresponding to each of the multiple processors according to the union of the masks; each processor among the multiple processors transmits the determined overall part to be updated corresponding to each of the multiple processors to at least one other processor, so that each processor among the multiple processors obtains the part to be updated of the target parameter corresponding to each of the multiple processors.

[0009] In a specific implementation, each of the multiple processors transmits the mask of the part to be updated between at least one other processor, so that each of the multiple processors obtains the mask of the part to be updated corresponding to each of the multiple processors, including: each of the multiple processors puts the mask of the part to be updated corresponding to each of the multiple processors into the cache of each processor; each of the multiple processors transmits the mask in the cache between at least one other processor, and puts the received mask into the cache of the processor, so that each of the multiple processors obtains the mask of the part to be updated corresponding to each of the multiple processors. The above steps can be implemented by a global aggregation (Allgather) algorithm.

[0010] In a specific implementation, each of the multiple processors transmits the determined corresponding overall part to be updated between at least one other processor, so that each of the multiple processors obtains the corresponding part to be updated of the target parameters of the multiple processors, including: each of the multiple processors puts the determined corresponding overall part to be updated into the cache of each processor; each of the multiple processors transmits the overall part to be updated in the cache between at least one other processor, and updates the part to be updated in the cache based on the received part to be updated. The above steps can be implemented by the Allreduce algorithm.

[0011] To implement the above implementation, the mask of each processor is first transmitted, and then each processor calculates the union of all masks, and then applies the union of masks to the tensor on its own processor to obtain the overall part to be updated, and then each processor transmits the obtained overall part to be updated. Therefore, after data transmission between multiple processors, the overall part to be updated of each processor can be obtained, and the overall part to be updated of each processor includes the part to be updated of its original target parameter, so that each processor obtains the part to be updated of the target parameter of all processors. During the entire transmission process, only the mask and the overall part to be updated are transmitted, and the communication volume is greatly reduced compared to the data transmission of the complete tensor. In addition, since the overall part to be updated on each processor is obtained by applying the same mask (i.e., the union of all masks) to the tensor in the processor, and the mask is used to indicate the position of the part to be updated in the tensor, the position of the overall part to be updated obtained by each processor using the union of masks applied to the tensor is aligned in the processor cache, so that after data transmission, there is no need to use Mask for decoding, thereby improving communication efficiency and improving the training speed of the NAS system.

[0012] In one possible implementation, multiple processors are divided into multiple groups, each of the multiple groups includes at least two processors with the same corresponding sub-network structure; each of the multiple processors transmits information on the to-be-updated part of the target parameter to be updated with at least one other processor, so that each of the multiple processors obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors, including: at least two processors in the same group transmit the to-be-updated part of the target parameter to be updated respectively, so that the first processor in the same group obtains the to-be-updated set in the group, wherein the to-be-updated set in the group includes the to-be-updated parts of the target parameter corresponding to all processors in the group; the first processors included in the multiple groups transmit the to-be-updated set in the group to each other, so that each of the multiple first processors obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors; each of the multiple first processors multicasts the to-be-updated part of the target parameter corresponding to each of the multiple processors, so that each processor in the same group obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors.

[0013] In the implementation method described above, since the NAS system is divided into multiple groups for gradient update, the subnetwork structure on the processors in each group is the same, so the intra-group communication can be performed first, so that each processor in the same group obtains the target parameter to be updated of other processors in the group, that is, the intra-group to be updated set, and then the first processor in each group performs inter-group communication, so that the first processor of each group obtains the intra-group to be updated set of the first processor of each other group, so that each first processor of each group obtains the target parameter to be updated corresponding to all processors, and finally, the first processor of each group can broadcast the target parameter to be updated corresponding to all processors, so that each processor in its own group obtains the target parameter to be updated corresponding to all processors. It can be understood that this makes full use of the characteristics of the NAS system that the intra-group subnetwork is shared and the inter-group subnetwork is not shared. The intra-group communication is performed first and then the inter-group communication is performed, so that when the inter-group communication is performed, the number of processors participating in the communication is greatly reduced (at this time, only the first processor of each group participates in the communication), thereby reducing the communication volume.

[0014] In a possible implementation, at least two processors in the same group transmit the part of the target parameter to be updated respectively, so that the first processor in the same group obtains the set to be updated in the group, including: each of the at least two processors in the same group puts the part of the target parameter to be updated into the cache of the processor; each of the at least two processors in the same group transmits the part to be updated in the cache to other processors, and updates the part to be updated in the cache of the processor based on the received part to be updated, so that the first processor in the same group obtains the set to be updated in the group. The first processors of multiple groups transmit the set to be updated in the group, so that each first processor of the multiple groups obtains the part to be updated of the target parameter corresponding to each of the multiple processors, including: each first processor encodes the set to be updated in the group to obtain the mask of the set to be updated in the group; the multiple first processors included in the multiple groups transmit the mask of the set to be updated in the group, so that each of the multiple first processors obtains the mask of the set to be updated in the group corresponding to each of the multiple first processors; each first processor obtains the part to be updated of the target parameter corresponding to each of the multiple processors based on the mask of the set to be updated in the group corresponding to each of the multiple first processors.

[0015] Optionally, each first processor obtains the to-be-updated parts of the target parameters corresponding to each of the multiple processors based on the masks of the to-be-updated sets within the group corresponding to each of the multiple first processors, including: the first processor obtains the union of the masks of the to-be-updated sets within the group corresponding to each of the multiple first processors based on the masks of the to-be-updated sets within the group corresponding to each of the multiple first processors; the first processor determines the overall to-be-updated set within the group corresponding to each of the multiple first processors according to the union of the masks of the to-be-updated sets within the group; and the determined overall to-be-updated set within the group is transmitted between the multiple first processors, so that each first processor obtains the to-be-updated parts of the target parameters corresponding to each of the multiple processors.

[0016] By implementing the above implementation, during the entire transmission process, only the mask and the set to be updated in the overall group (including the part to be updated of the target parameter) are transmitted between processors. Compared with transmitting the complete tensor data, the communication volume is greatly reduced. In addition, after the data transmission, there is no need to use the mask for decoding, thereby improving the communication efficiency and the training speed of the NAS system.

[0017] Optionally, each first processor obtains the to-be-updated parts of the target parameters corresponding to each of the multiple processors based on the masks of the to-be-updated sets within the group corresponding to each of the multiple first processors, including: transmitting the to-be-updated sets within the group between the multiple first processors so that each first processor obtains the to-be-updated sets within the group of the multiple first processors; and the multiple first processors decode the to-be-updated sets within the group of the multiple first processors according to the masks of the to-be-updated sets within the group corresponding to each of the multiple first processors to obtain the to-be-updated parts of the target parameters corresponding to each of the multiple processors.

[0018] By implementing the above implementation method, during the entire transmission process, only the MASK and the part of the target parameter to be updated are transmitted. Compared with other methods (transmitting the complete tensor for data), the communication volume is greatly reduced, thereby improving communication efficiency and increasing the training speed of the NAS system.

[0019] In a second aspect, a data processing device is provided, which is applied to a neural network structure search NAS system. The system includes multiple processors, each processor is used to train a subnetwork in the search supernetwork, and the device includes: a training unit, the training unit is used to perform current iterative training on the corresponding subnetwork through each processor of the multiple processors, and obtain the part to be updated of the target parameter of each processor; the transmission unit is used to transmit the information of the part to be updated of the target parameter obtained between each processor of the multiple processors and at least one other processor, so that each processor of the multiple processors obtains the part to be updated of the target parameter corresponding to each of the multiple processors.

[0020] In one embodiment, the device further includes an encoding unit, which is used to encode the part to be updated of the target parameter through each processor to obtain a mask of the part to be updated; the information of the part to be updated includes: the part to be updated and the mask of the part to be updated.

[0021] In one embodiment, a transmission unit is used to transmit a mask of a part to be updated between each of a plurality of processors and at least one other processor, so that each of the plurality of processors obtains a mask of a part to be updated corresponding to each of the plurality of processors; the transmission unit is used to obtain a union of masks of parts to be updated corresponding to each of the plurality of processors through each of the plurality of processors; the transmission unit is used to determine, through each of the plurality of processors, an overall part to be updated corresponding to each of the plurality of processors according to the union of masks; the transmission unit is used to transmit the determined overall part to be updated corresponding to each of the plurality of processors between each of the plurality of processors and at least one other processor, so that each of the plurality of processors obtains an updated part of a target parameter corresponding to each of the plurality of processors.

[0022] In one embodiment, a transmission unit is used to place the mask of the corresponding part to be updated into the cache of each processor through each of the multiple processors; the transmission unit is used to transmit the mask in the cache between each of the multiple processors and at least one other processor, and place the received mask into the cache of the processor, so that each of the multiple processors obtains the mask of the corresponding part to be updated of the multiple processors.

[0023] In one embodiment, the transmission unit is used to place the corresponding overall part to be updated determined by the transmission into the cache of each processor through each of the multiple processors; the transmission unit is used to transmit the overall part to be updated in the cache between each of the multiple processors and at least one other processor, and update the part to be updated in the cache based on the received part to be updated.

[0024] In one embodiment, multiple processors are divided into multiple groups, each of the multiple groups includes at least two processors with the same corresponding sub-network structure; the transmission unit is used to transmit the to-be-updated parts of the target parameters between at least two processors in the same group, so that the first processor in the same group obtains the to-be-updated set in the group, wherein the to-be-updated set in the group includes the to-be-updated parts of the target parameters corresponding to all processors in the group; the transmission unit is used to transmit the to-be-updated set in the group between the first processors included in the multiple groups, so that each first processor of the multiple first processors obtains the to-be-updated parts of the target parameters corresponding to the multiple processors; the transmission unit is used to multicast the to-be-updated parts of the target parameters corresponding to the multiple processors through each first processor of the multiple first processors, so that each processor in the same group obtains the to-be-updated parts of the target parameters corresponding to the multiple processors.

[0025] In one embodiment, the transmission unit is used to place the to-be-updated part of the target parameter into the cache of the processor through each of at least two processors in the same group; the transmission unit is used to transmit the to-be-updated part in the cache between each of at least two processors in the same group and other processors, and update the to-be-updated part in the cache of the processor based on the received to-be-updated part, so that the first processor in the same group obtains the to-be-updated set within the group.

[0026] In one embodiment, the encoding unit is used to encode the set to be updated in the group through each first processor to obtain the mask of the set to be updated in the group; the transmission unit is used to transmit the mask of the set to be updated in the group between multiple first processors included in multiple small groups, so that each of the multiple first processors obtains the mask of the set to be updated in the group corresponding to each of the multiple first processors; the transmission unit is used to obtain the to-be-updated parts of the target parameters corresponding to each of the multiple processors based on the masks of the set to be updated in the group corresponding to each of the multiple first processors through each first processor.

[0027] In one embodiment, a transmission unit is used to obtain, through a first processor, a union of masks of sets to be updated in a group corresponding to each of the multiple first processors based on masks of sets to be updated in a group corresponding to each of the multiple first processors; the transmission unit is used to determine, through the first processor according to the union of masks of sets to be updated in the group, an overall set to be updated in a group corresponding to each of the multiple first processors; the transmission unit is used to transmit the determined overall set to be updated in a group between the multiple first processors, so that each first processor obtains the to-be-updated portion of the target parameters corresponding to each of the multiple processors.

[0028] In one embodiment, a transmission unit is used to transmit the set to be updated within the group between multiple first processors, so that each first processor obtains the set to be updated within the group of multiple first processors; the transmission unit is used to decode the set to be updated within the group of multiple first processors according to the masks of the set to be updated within the group corresponding to each of the multiple first processors through the multiple first processors, so as to obtain the to-be-updated parts of the target parameters corresponding to each of the multiple processors.

[0029] According to a third aspect, a computer program product is provided, comprising a computer program, which implements the method described in the first aspect when the computer program is read and executed by a computing device.

[0030] In a fourth aspect, a computer-readable storage medium is provided, comprising instructions, which, when executed on a computing device, enable the computing device to implement the method described in the first aspect.

[0031] In a fifth aspect, an electronic device is provided, including a processor and a memory, wherein the processor executes a code in the memory to implement the method described in the first aspect.

[0032] In a sixth aspect, a chip is provided, comprising a memory and a processor; the memory is coupled to the processor, the processor comprises a modem processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and the processor reads computer instructions from the memory so that the chip executes the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0034] Figure 1 It is a schematic diagram of the structure of a neural network structure search system;

[0035] Figure 2 It is a flowchart of a data processing method provided by this application;

[0036] Figure 3 It is a flowchart of a data processing method provided by this application;

[0037] Figure 4 It is a flowchart of a data processing method provided by this application;

[0038] Figure 5 It is a schematic diagram of the structure of multiple GPUs in a ring architecture;

[0039] Figure 6 It is a flow chart of a data processing method provided by the present application;

[0040] Figure 7 It is a flow chart of a data processing method provided by the present application;

[0041] Figure 8 is a schematic diagram of the structure of a GPU provided by this application;

[0042] Fig. 9 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] First, the application scenario applicable to this application, neural network structure search (NAS), is explained.

[0045] With the emergence of large-scale data sets and the rapid development of computing power (such as the computing performance of NVIDIA GPU), deep learning, as one of the important algorithms in the field of artificial intelligence, has begun to show its amazing dominance in tasks such as data classification, object segmentation, and object detection. In particular, the emergence of AlphaGo in 2016 has allowed people around the world to see the huge potential of artificial intelligence, and has also made people have beautiful visions for advanced technologies such as unmanned driving, remote surgery, and scene understanding. In recent years, the development of deep learning has gradually subverted the era of traditional machine learning feature engineering and pushed the wave of artificial intelligence to its highest point in history. In deep learning, the structure of the neural network relies on manual design, and researchers need to spend a lot of time and energy to design and improve the model structure according to the task objectives. For example, the residual network (Resnet) and other network structures that shine in image classification tasks are all the result of researchers' careful design over the years.

[0046] In order to solve the problem that the neural network structure is completely dependent on manual design, researchers began to try to gradually transform the design of the neural network structure into machine automatic design, so Neural Architecture Search (NAS) came into being. NAS refers to finding a better network structure in a given search space and trying to optimize the weight parameters of the network structure to obtain a trained network model. NAS usually uses a recurrent neural network (RNN) as a controller to generate multiple sub-networks according to the topological structure of the target neural network required by the task (for example, a convolutional neural network can be divided into a convolutional layer and a pooling layer, etc., and the controller can generate multiple convolutional layer sub-networks, and then generate multiple pooling layer sub-networks, etc.), and then train and evaluate each sub-network. If the network performance (such as accuracy) of the sub-network is not good, the structure will be discarded, and the remaining sub-networks will be spliced ​​into an optimal network structure. Since the weights of each sub-network of NAS are obtained through independent training, that is, each sub-network requires a complete training process, which consumes huge computing resources and time. Therefore, NAS often distributes the workload to multiple computing nodes containing one or more processors (usually graphics processing units (GPUs)) to accelerate training. The following example illustrates a multi-machine and multi-GPU NAS training system.

[0047] Figure 1 This is a schematic diagram of the structure of a NAS system. Figure 1 As shown, the NAS system may include X( Figure 1 Taking X=2 as an example, where X is a positive integer), there are 10 groups 110, each of which may include multiple processors 130, each of which may use the tensors on the processor to iteratively train the sub-network structure, and the processors 130 may exchange model parameters or gradients through the bus 140 in each iteration until the optimal network structure is selected and the weight parameters of the network structure are optimized to obtain a trained network model. Here, the processor 130 may be a GPU or other processors. In order to make the present application better understood, the following examples will uniformly illustrate the processor 130 as a GPU.

[0048] Each group 110 uses a subnet of the same structure for training. Different groups 110 use different subnet structures. The union of the subnet structures of all groups 110 is a supernet structure. Therefore, if there are X groups 110, it means that the system includes X subnet structures. For example, Figure 1Group 1 and Group 2 in the training use different sub-network structures (Group 1 uses Structure 1, Group 2 uses Structure 2). Multiple GPUs in the same group 110 use the same sub-network structure, but the tensors (Tensor) can be the same or different. The tensor can be the input image size, output image size, moving step size, pooling layer size, convolution kernel size and other parameters of each layer of the network.

[0049] For example, Figure 1 Group 1 in the example includes GPU1, GPU2, GPU5, and GPU6, where GPU1 and GPU2 use tensor 1 to train structure 1, and GPU5 and GPU6 use tensor 2 to train structure 1; Group 2 includes GPU3, GPU4, GPU7, and GPU8, where GPU3 and GPU4 use tensor 1 to train structure 2, and GPU7 and GPU8 use tensor 2 to train structure 2. It should be understood that Figure 1 For illustration only, this application does not limit the number of groups 110, the number of GPUs in each group 110, the number of sub-network structures, and the number of tensors.

[0050] In a specific implementation, multiple sub-network structures are generated according to the task objectives (for example, a face recognition model needs to be trained), such as designing some sub-network structures of convolutional layers and pooling layers according to the topological structure of the convolutional neural network; then each GPU trains these sub-network structures on the training set, and each GPU130 can use the training set to train sub-network structures with different structures and tensors; then all GPUs communicate with each other and exchange trained gradient data or model parameters, so that each GPU obtains the gradient data or model parameters calculated by other GPUs to update the weights of the entire network, and the above steps are repeated until a trained network model is obtained.

[0051] In summary, in the NAS system, GPUs need to exchange model parameters or gradients during each iteration, which consumes a lot of communication overhead, greatly limiting the NAS training speed. However, after each GPU 130 uses the tensor on the processor to train the sub-network structure, only a small part of the data on the tensor may need to be updated, such as Figure 1 The shaded portion of each GPU 130 in FIG. 1 is the tensor to be updated 150 on each GPU. Therefore, when exchanging model parameters or gradient data between multiple GPUs in a NAS system, a large amount of communication redundancy will be generated, such as Figure 1 The white parts in the figure are actually data that do not need to be updated, which limits the speed of NAS training neural networks.

[0052] Therefore, in order to solve the problem of a large amount of communication redundancy in the above-mentioned NAS system, the present application provides a data processing method, which is applied to the above-mentioned NAS system. The GPU in this method only transmits the part to be updated of each tensor during communication with other GPUs. Compared with the above-mentioned method of directly transmitting all the data on the GPU (that is, the entire trained tensor), the communication volume can be greatly reduced and the training speed of the NAS system can be improved.

[0053] The following is combined with Figure 2-Figure 3 , the data processing method provided in this application is described in detail.

[0054] The present application provides a data communication method, which is applied to a NAS system. The structure of the NAS system can be as follows: Figure 1 As shown. Figure 2 As shown, the method may include the following steps:

[0055] S210: Each of the multiple GPUs performs current iterative training on the corresponding sub-network structure to obtain the to-be-updated portion of the target parameter of each GPU, wherein the target parameter is the tensor of the aforementioned content, and the to-be-updated portion of the target parameter is the to-be-updated data of the corresponding tensor on each GPU.

[0056] Specifically, each of the multiple GPUs corresponds to a target parameter and a sub-network structure, and GPUs in the same group correspond to the same sub-network structure, and the union of the sub-network structures is the neural network to be trained by the NAS system. Figure 3 For example, GPU1 and GPU2 in group 1 can train sub-network structure 1 including target parameter 1, and GPU3 and GPU4 in group 2 can train sub-network structure 2 including target parameter 1. It can be understood that after each GPU trains the target parameters and sub-networks on the processor, each GPU can obtain a part of the target parameters to be updated, such as Figure 1 The white rectangular box of GPU1 is the part of the target parameters of GPU1 to be updated, and the dark rectangular box of GPU2 is the part of the target parameters of GPU2 to be updated, and examples are not given one by one here.

[0057] S220: Each processor among the multiple processors transmits the obtained information of the to-be-updated part of the target parameter to at least one other processor, so that each processor obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors.

[0058] Still Figure 3 For example, Figure 3As shown on the left, although GPU1 and GPU2 have the same subnetwork structure, after training, the positions of the target parameters to be updated of GPU1 and GPU2 are the same, but the updated data are different. Similarly, the target parameters to be updated of GPU3 and GPU4 are also different. Figure 3 The data of the to-be-updated part of the target parameter of each GPU is represented by different colors. After step S210 to step S220, Figure 3 As shown on the right, GPU1, GPU2, GPU3 and GPU4 obtain the to-be-updated parts 1 to 4 of the target parameters corresponding to the multiple processors. Figure 3 The target parameters to be updated of GPU1, GPU2, GPU3 and GPU4 after communication are represented by the same color. It should be understood that Figure 3 It is for illustration only and does not constitute a specific limitation.

[0059] In a specific implementation, the information of the part to be updated of the above-mentioned target parameter may include the part to be updated and the mask of the part to be updated. The part to be updated may be the specific value of the part to be updated of the target parameter, and the Mask may be obtained after each processor encodes the part to be updated of the target parameter. The Mask is used to indicate the position of the part to be updated of the target parameter in the target parameter, wherein the mask represents masking, and the code represents the value in the tensor. The mask is generally a string of binary digits with only 1 and 0 elements, indicating that the corresponding position is masked or not. Whether 0 or 1 represents masking can be defined by oneself. The role of the mask is to mask part of the value in the target parameter, thereby achieving the purpose of indicating the position of the part to be updated of the target parameter in the target parameter. For example, the information of the part to be updated of the target parameter of each GPU includes: the part to be updated, that is, Figure 3 The data value marked by the dark part in the tensor; and the Mask of the part to be updated, which is used to indicate the position of the data value to be updated in the entire tensor, that is, the position of the dark part of the rectangular box in the tensor. It should be understood that the above examples are for illustration and cannot constitute a specific limitation.

[0060] In the specific implementation, the corresponding algorithm can be selected according to the actual situation to obtain the updated part of the target parameter corresponding to each processor and the corresponding Mask. The following takes the NAS training of the convolutional neural network as an example to illustrate how each processor obtains the updated part of the corresponding target parameter and the corresponding Mask.

[0061] It is understandable that the network structure layers that need parameter update in CNN are only convolutional layers, fully connected layers, and batch normalization layers (BN layers). Therefore, we can only obtain the updated parts of the target parameters that need to be updated in these three layers of the network, and then determine the MASK corresponding to the updated parts of the target parameters, so as to obtain the updated parts of the target parameters and the corresponding MASK on each GPU.

[0062] First, obtain the target parameters of the convolution layer, fully connected layer, and batch normalization layer to be updated. The parameters of the convolution layer include convolution kernel parameters and additional parameters. The additional parameters are parameters that do not need to be updated, such as stride, number of groups, padding, etc.; convolution kernel parameters are parameters that need to be updated, such as (Cout, Cin, k, k), where Cin and Cout are the size of the input and output channels, respectively, and k is the length and width of the convolution kernel. When performing structural search, we often use S1 = (groups, out_grouch, in_grouch, kernel_size) to determine the parameters of the convolution kernel (output space). Suppose the search space in the training process is S0 = (max_groups, max_out_grouch, max_in_grouch, max_kernel_size), so as to obtain the convolution kernel parameters that need to be updated, such as weight and bias.

[0063] Similarly, since the parameters that need to be updated in the fully connected layer include Cout and Cin, assume that the parameters used to determine the fully connected layer during structure search are S1 = (out_features, in_features), and the search space is S0 = (max_out_features, max_in_features), so as to obtain the fully connected layer parameters (weight and bias) that need to be updated.

[0064] Similarly, since the parameters that need to be updated in the batch normalization layer include num_features, running_mean and running_var, it is assumed that the parameters used to determine the batch normalization layer during the structure search are S1 = (num_features) and the search space is S0 = (max_num_features), so as to obtain the fully connected layer parameters that need to be updated.

[0065] Finally, after obtaining the to-be-updated part of the target parameter corresponding to each GPU, a string of binary codes is used to perform a bitwise AND operation on the to-be-updated part of each tensor, and the Mask corresponding to each to-be-updated part is obtained by masking the current input bit. It should be understood that the specific method of obtaining the to-be-updated part of the target parameter and its corresponding Mask is only for illustration and does not constitute a specific limitation.

[0066] The present application provides multiple implementation methods of implementing step S220 according to the above-mentioned Mask, which are explained one by one below.

[0067] (I) Method based on Mask union

[0068] In one embodiment, if Figure 4 As shown, the above step S220 may include the following steps:

[0069] S221A: Each of the multiple processors transmits the Mask of the part to be updated to at least one other processor, so that each of the multiple processors obtains the Mask of the part to be updated corresponding to each of the multiple processors.

[0070] In one embodiment, after obtaining the MASK of each processor, the specific steps of transmitting the MASK between multiple processors so that each processor obtains the MASK of other processors may be as follows: Figure 4 As shown, each processor puts the mask of the corresponding part to be updated into the cache of each processor, and each processor transmits the mask in the cache to other processors, and puts the received mask into the cache of the processor, so that each processor obtains the mask of the corresponding part to be updated of all processors.

[0071] In a specific implementation, multiple processors can be in a ring structure for data transmission. The M GPUs in the group are arranged in a logical ring, and each GPU has a left neighbor and a right neighbor, for example Figure 5 As shown, Figure 5 The three GPUs in the system, namely GPU1, GPU2 and GPU3, can form a logical ring. GPU1 can send data in GPU1 cache to GPU2 on the right, namely MASK1; GPU2 can send data in GPU2 cache to GPU3 on the right, namely MASK2; GPU3 can send data in GPU3 cache to GPU1 on the right, namely MASK3. Of course, multiple processors can also perform data transmission in a centralized structure. For example, GPU1 is the main processor, and all GPUs send their own MASKs to GPU2, and GPU1 then sends all processors' MASKs to each GPU one by one. In order to facilitate a better understanding of this application, the following will be unified by processor. Figure 5 The ring structure shown is used as an example for data transmission.

[0072] Furthermore, the cache of each GPU can be evenly divided into multiple cache blocks. When GPUs exchange data in the cache according to a logical ring, they can exchange according to cache blocks. In simple terms, each time the MASK in the cache is exchanged between GPUs, only one cache block is exchanged. Multiple cache blocks are accumulated to form MASK1, MASK2 and MASK3. In this way, the ring structure continuously exchanges data in the cache, and eventually the cache of each GPU contains MASK1, MASK2 and MASK3.

[0073] Specifically, if the amount of data cached in each GPU is large, the cache in each GPU can be divided into multiple cache blocks. The above process can be implemented through the global aggregation (Allgather) algorithm. In this algorithm, the GPU will perform multiple rounds of data exchange. In each round of data exchange, each GPU sends a block to its right neighbor and receives a block from its left neighbor to cover the cache block. In each round of data exchange, each GPU chooses to send and receive different blocks, so that each GPU ultimately contains the final values ​​of all types of cache blocks.

[0074] For example, suppose Figure 5 The data distribution on GPU1 to GPU3 is shown in Table 1: The cache data on the three GPUs is evenly divided into three blocks, MASK1 on GPU1 is represented by three B-type blocks, MASK2 on GPU2 is represented by three C-type blocks, and MASK3 on GPU3 is represented by three A-type blocks:

[0075] Table 1 Cache block distribution table of GPU1, GPU2 and GPU3

[0076] GPU1 / B1+B3+B2 / GPU2 / / C2+C1+C3 GPU3 A3+A2+A1 / /

[0077] Assuming that during the first round of data exchange, GPU1 only transmits class B blocks to the right, GPU2 only transmits class C blocks to the right, and GPU3 only sends class A blocks to the right, then after the first round of data exchange, the cache block distribution of GPU1, GPU2, and GPU3 can be shown in Table 2:

[0078] Table 2 Cache block distribution after the first round of data exchange in the Allgather phase of GPU1, GPU2 and GPU3

[0079] GPU1 A3+A2+A1 B1+B3+B2 / GPU2 / B1+B3+B2 C2+C1+C3 GPU3 A3+A2+A1 / C2+C1+C3

[0080] Assuming that in the second round of data exchange, GPU1 only sends class A blocks, GPU2 only sends class B blocks, and GPU3 only sends class C blocks, then after the second round of data exchange, the cache block distribution of GPU1, GPU2, and GPU3 can be shown in Table 3:

[0081] Table 3 Cache block distribution table after the second round of data exchange for GPU1, GPU2 and GPU3

[0082] GPU1 A3+A2+A1 B1+B3+B2 C2+C1+C3 GPU2 A3+A2+A1 B1+B3+B2 C2+C1+C3 GPU3 A3+A2+A1 B1+B3+B2 C2+C1+C3

[0083] At this time, GPU1, GPU2 and GPU3 all obtain data on other cards. The above examples are for illustration only and do not constitute specific limitations.

[0084] S222A: Each of the multiple processors obtains a union of masks of the to-be-updated parts corresponding to the multiple processors.

[0085] Understandable, such as Figure 5 As shown, after each processor obtains the masks of other processors, it can take the union of the masks of other processors and the mask of its own processor to obtain the union of the masks of the to-be-updated parts corresponding to the multiple processors.

[0086] S223A: Each of the multiple processors determines the overall to-be-updated portion corresponding to each of the multiple processors according to the union of the masks.

[0087] It can be understood that each GPU can obtain an overall part to be updated according to the union of the masks and the target parameters, and the overall part to be updated includes the part to be updated of the target parameters of the GPU.

[0088] For example, if the target parameter (i.e., tensor) of GPU1 is X, the part to be updated of the target parameter is Y, and the mask is M, then XM=Y. After steps S221A to S222A, the union of the masks is M'. At this time, according to the tensor X on GPU1, the overall part to be updated can be obtained as Y'=XM'. It can be understood that the union of the masks M' is greater than the mask M, so the overall part to be updated includes the part Y to be updated of the target parameter. The above examples are for illustration only and are not specifically limited in this application.

[0089] It is worth noting that, referring to the foregoing content, Mask is used to indicate the position of the part to be updated in the target parameter. At step S223A, each GPU obtains its own overall part to be updated according to the same mask (i.e., the union of the masks). Therefore, although the data of the overall part to be updated obtained by each GPU is different, the position in the cache is the same, for example Figure 5 shown.

[0090] S224A: Each of the multiple processors transmits the determined corresponding overall to-be-updated parts to at least one other processor, so that each of the multiple processors obtains the to-be-updated parts of the target parameters corresponding to the multiple processors.

[0091] In one embodiment, each processor transmits the union of the parts to be updated with other processors so that each of the multiple processors obtains the parts to be updated of the target parameters corresponding to each of the multiple processors, including: each of the multiple processors puts the parts to be updated into the cache of each processor; each of the multiple processors transmits the parts to be updated in the cache with at least one other processor, and updates the parts to be updated in the cache based on the received parts to be updated.

[0092] In a specific implementation, the above process can be implemented through the global reduction (Allreduce) algorithm, which can include two stages. The first stage is the scatter-Reduce stage. In this stage, the GPU will perform multiple rounds of data exchange. In each round of data exchange, the GPU sends a block to its right neighbor and receives a block from its left neighbor and accumulates it in the cache block. In each round of data exchange, each GPU chooses to send and can receive different blocks, so that each GPU eventually contains the final value of a class of blocks. The second stage is the Allgather stage mentioned above. In this stage, the GPU will also perform multiple rounds of data exchange, but in each round of data exchange, the GPU sends a block to its right neighbor and receives a block from its left neighbor to cover the cache block. In each round of data exchange, each GPU chooses to send and can receive different blocks, so that each GPU eventually contains the final value of all types of cache blocks.

[0093] For example, as shown in Table 4, the cache blocks of GPU1 include A1, B1 and C1 (the overall part to be updated of GPU1), the cache blocks of GPU2 include A2, B2 and C2 (the overall part to be updated of GPU2), and the cache blocks of GPU3 include A3, B3 and C3 (the overall part to be updated of GPU3).

[0094] Table 4 Initial cache block distribution table of GPU1, GPU2 and GPU3

[0095] GPU1 A1 B1 C1 GPU2 A2 B2 C2 GPU3 A3 B3 C3

[0096] Assuming that in the first round of data exchange, GPU1 only sends class A blocks, GPU2 only sends class B blocks, and GPU3 only sends class C blocks, then after the first round of data exchange, the cache block distribution of GPU1, GPU2, and GPU3 can be shown in Table 5:

[0097] Table 5 Cache block distribution after the first round of data exchange in the scatter-Reduce phase of GPU1, GPU2 and GPU3

[0098]

[0099]

[0100] Assuming that during the second round of data exchange, GPU1 only sends C-type blocks, GPU2 only sends A-type blocks, and GPU3 only sends B-type blocks, then after the second round of data exchange, the cache distribution of GPU1, GPU2, and GPU3 can be shown in Table 6:

[0101] Table 6 Cache block distribution table after the scatter-Reduce phase of GPU1, GPU2 and GPU3

[0102] GPU1 A1 B1+B3+B2 C1+C3 GPU2 A2+A1 B2 C2+C1+C3 GPU3 A3+A2+A1 B3+B2 C3

[0103] In this way, the scatter-reduce stage is completed. After the first stage, the cache block of GPU1 will include a complete class B block, the cache block of GPU2 will include a complete class C block, and the cache block of GPU3 will include a complete class A block. Then we can enter the second stage, the Allgather stage. During the communication process, each GPU will overwrite each cache block it receives with its own cache block. Assuming that in the first round of data exchange, GPU1 only sends class B blocks, GPU2 only sends class C blocks, and GPU3 only sends class A blocks, then after the first round of data exchange, the cache block distribution of GPU1, GPU2, and GPU3 can be shown in Table 7:

[0104] Table 7 Cache block distribution table after the first round of data exchange in the Allgather phase of GPU1, GPU2 and GPU3

[0105] GPU1 A3+A2+A1 B1+B3+B2 C1+C3 GPU2 A2+A1 B1+B3+B2 C2+C1+C3 GPU3 A3+A2+A1 B3+B2 C2+C1+C3

[0106] Assuming that during the second round of data exchange, GPU1 only sends class A blocks, GPU2 only sends class B blocks, and GPU3 only sends class C blocks, then after the second round of data exchange, the cache block distribution of GPU1, GPU2, and GPU3 can be shown in Table 8:

[0107] Table 8 Cache block distribution table after the Allgather phase of GPU1, GPU2 and GPU3 ends

[0108] GPU1 A3+A2+A1 B1+B3+B2 C2+C1+C3 GPU2 A3+A2+A1 B1+B3+B2 C2+C1+C3 GPU3 A3+A2+A1 B1+B3+B2 C2+C1+C3

[0109] In this way, the Allgather stage is completed, and GPU1, GPU2 and GPU3 all obtain the data on all GPUs. It should be understood that the above examples are for illustration only and cannot constitute a specific limitation.

[0110] It can be understood that in the above steps S221A-S224A, the MASK of each GPU is first transmitted, and then each GPU calculates the union of all MASKs, and then applies the union of MASKs to the tensor on its own GPU to obtain the overall part to be updated, and then each GPU transmits the obtained overall part to be updated. Therefore, after data transmission between multiple GPUs, the overall part to be updated of each GPU can be obtained, and the overall part to be updated of each GPU includes the part to be updated of its original target parameters, so that each GPU obtains the part to be updated of the target parameters of all GPUs. During the entire transmission process, only the MASK and the overall part to be updated are transmitted, which greatly reduces the communication volume compared to other methods (transmitting complete tensors). Moreover, since the overall part to be updated on each GPU is obtained by applying the same mask (i.e., the union of all MASKs) to the tensor in the processor, and the Mask is used to indicate the position of the part to be updated in the tensor, the position of the overall part to be updated obtained by each GPU using the union of Masks applied to the tensor in the processor cache is aligned. In this way, after data transmission, there is no need to use Mask for decoding, thereby improving communication efficiency and increasing the training speed of the NAS system.

[0111] (II) Packet transmission based method

[0112] Understandably, due to the NAS system (such as Figure 1 As shown in FIG. 1 , the GPUs in each group are divided into multiple groups for gradient update. The subnetwork structure on the GPUs in each group is the same. Therefore, intra-group communication can be performed first, so that each GPU in the same group obtains the target parameters of other GPUs in the group to be updated, that is, the intra-group set to be updated. Then, the first processor in each group performs inter-group communication, so that the first processor of each group obtains the intra-group set to be updated of the first processors of each other group, so that each first processor of each group obtains the target parameters corresponding to all processors. Finally, the first processor of each group can broadcast the target parameters corresponding to all processors to be updated, so that each processor in its own group obtains the target parameters corresponding to all processors. It can be understood that this makes full use of the characteristics of intra-group subnetwork sharing and inter-group subnetwork non-sharing of the NAS system. Intra-group communication is performed first and then inter-group communication is performed, so that when inter-group communication is performed, the number of processors participating in the communication is greatly reduced (at this time, only the first processor of each group participates in the communication), thereby reducing the amount of communication.

[0113] Therefore, in one embodiment, if Figure 6 As shown, the above step S220 may further include the following steps:

[0114] S221B: At least two processors in the same group transmit the parts of the target parameters to be updated respectively, so that the first processor in the same group obtains the set to be updated in the group, wherein the set to be updated in the group includes the parts of the target parameters to be updated corresponding to all processors in the group.

[0115] In one embodiment, since the target parameters (tensors) in the GPUs in the same group in the NAS system are the same and have the same structure, although the data of the to-be-updated part of the target parameters of the GPUs in each group are different, the position of the to-be-updated part of the target parameters in the target parameters is the same, and MASK is used to indicate the position of the to-be-updated part of the target parameters in the target parameters. Therefore, the MASK corresponding to the to-be-updated part of the GPUs in the group is the same, and MASK transmission is not required during intra-group communication. The GPUs in each group can put the to-be-updated part of their own target parameters into the processor's cache and then communicate within the group.

[0116] Specifically, each of the at least two processors in the same group puts the to-be-updated part of the target parameter into the processor's cache, and then each of the at least two processors in the same group transmits the to-be-updated part in the cache to other processors, and updates the to-be-updated part in the processor's cache based on the received to-be-updated part, so that the first processor in the same group obtains the to-be-updated set in the group. In a specific implementation, the steps of the above-mentioned intra-group communication can refer to the embodiments of Tables 4 to 8 of the aforementioned content, and the intra-group communication is implemented using the Allreduce algorithm, which will not be repeated here.

[0117] S222B: The first processors respectively included in the multiple groups transmit the set to be updated within the group, so that each of the multiple first processors obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors. The first processor may be the default processor in each group, such as the first processor in each group, or the processor with the lowest cache occupancy rate in the group, which is not specifically limited in this application.

[0118] S223B: The first processor of each group broadcasts the to-be-updated parts of the target parameters corresponding to all the processors within the group, so that each processor in the group obtains the to-be-updated parts of the target parameters corresponding to all the processors.

[0119] For example Figure 6As shown, first, GPU1 and GPU2 of group 1 communicate within the group, so that GPU1 and GPU2 in group 1 obtain the to-be-updated sets 1 to 2 in the group, and GPU3 and GPU4 in group 2 communicate within the group, so that GPU3 and GPU4 in group 2 obtain the to-be-updated sets 3 to 4 in the group; secondly, group 1 and group 2 communicate between groups, GPU1 in group 1 acts as the first processor, and GPU2 in group 2 acts as the first processor, and GPU1 and GPU2 communicate between groups, so that GPU1 and GPU2 obtain the to-be-updated parts 1 to 4 of the target parameters corresponding to all the processors respectively; finally, each first processor multicasts to all the processors in the group, GPU1 sends the to-be-updated parts of the target parameters corresponding to all the processors respectively obtained to GPU2 in the group, and GPU3 sends the to-be-updated parts of the target parameters corresponding to all the processors respectively obtained to GPU4 in the group, thereby completing this data communication.

[0120] The present application provides two methods to implement the inter-group communication described in step S222B above. The first method will be described in steps S2221B to S2223B below, and the second method will be described in steps S2224B to S2227B below. The two inter-group communication methods are described below.

[0121] In one embodiment, inter-group transmission may be implemented by the method based on Mask union in the foregoing content. Specifically, step S222B may include the following steps:

[0122] S2221B: Each first processor performs encoding processing on the set to be updated in the group to obtain a mask of the set to be updated in the group; the masks of the set to be updated in the group are transmitted between the multiple first processors included in the multiple small groups, so that each of the multiple first processors obtains the mask of the set to be updated in the group corresponding to each of the multiple first processors;

[0123] S2222B: Each of the multiple first processors obtains a union of masks of the sets to be updated in the group corresponding to each of the multiple first processors; each of the multiple first processors determines the overall set to be updated in the group corresponding to each of the multiple first processors according to the union of masks of the sets to be updated in the group;

[0124] S2223B: The determined set to be updated in the overall group is transmitted between the multiple first processors, so that each first processor obtains the to-be-updated part of the target parameters corresponding to each of the multiple processors.

[0125] In simple terms, the first processor of each group first generates a mask of the set to be updated within the group, and then multiple first processors exchange the masks of the set to be updated within the group through the Allgather algorithm, so that all first processors obtain the masks of the sets to be updated corresponding to all groups of first processors, and then calculate the union of the masks of the sets to be updated corresponding to each of the first processors, and each first processor applies the calculated union of the masks to the tensor of its own processor to obtain the overall set to be updated in the group of each first processor, and the overall set to be updated in the group includes the set to be updated corresponding to the first processor, and then each first processor exchanges its corresponding overall set to be updated in the group through the allreduce algorithm, so that each first processor obtains the overall set to be updated in the group of all first processors. The description of the steps not expanded in the above step flow can refer to the aforementioned Figure 4 Steps S221A-S224A of the embodiment are not repeated here.

[0126] It should be noted that after each first processor obtains the overall in-group set to be updated of all first processors, the first processor of each small group can propagate the obtained overall in-group set to be updated of all first processors within the group, so that each processor in its own group obtains the overall in-group set to be updated of all first processors, and the overall in-group set to be updated of all first processors includes the to-be-updated parts of the target parameters corresponding to all processors, thereby achieving the purpose of each of the multiple processors obtaining the to-be-updated parts of the target parameters corresponding to the multiple processors.

[0127] When using the above method for inter-group transmission, only the MASK and the set to be updated in the overall group are transmitted during the entire transmission process. Compared with other methods that transmit the complete tensor for data transmission, the communication volume is greatly reduced. In addition, after data transmission, there is no need to use the Mask for decoding, thereby improving communication efficiency and increasing the training speed of the NAS system.

[0128] In one embodiment, inter-group communication can also be achieved by another method, such as Figure 7 As shown, step S222B may include the following steps:

[0129] S2224B: Each first processor performs encoding processing on the set to be updated in the group to obtain a mask of the set to be updated in the group, such as Figure 7 As shown, assuming that the first processor of group 1 is GPU1, the first processor of group 2 is GPU3, and the first processor of group 3 is GPU9, then GPU1 can encode the sets 1 to 2 to be updated in group 1 to obtain the mask of the sets to be updated in group 1, that is, group 1 MASK. Similarly, GPU3 can obtain group 2 MASK, and GPU9 can obtain group 3 MASK.

[0130] S2225B: The masks of the to-be-updated sets in the groups are transmitted between the first processors of the multiple groups, so that each of the multiple first processors obtains the masks of the to-be-updated sets in the groups corresponding to the multiple first processors. Figure 7 As shown, GPU1, GPU3 and GPU9 transmit the masks of the to-be-updated sets in their respective groups, so that the caches of GPU1, GPU3 and GPU9 all include group 1 MASK, group 2 MASK and group 3 MASK.

[0131] S2226B: The plurality of first processors of the plurality of groups transmit the to-be-updated sets within the group to each other, so that each first processor obtains the to-be-updated sets within the group of the plurality of first processors, such as Figure 7 As shown, GPU1, GPU3 and GPU9 transmit the masks of their respective in-group to-be-updated sets, so that the caches of GPU1, GPU3 and GPU9 all include in-group to-be-updated set 1, in-group to-be-updated set 2 and in-group to-be-updated set 3. Figure 7 Dark rectangles marked with 1 to 10 represent the in-group set 1 to be updated, the in-group set 2 to be updated, and the in-group set 3 to be updated.

[0132] It should be noted that step S2225B and step S2226B may be performed simultaneously or sequentially, which may be determined based on the processing capability of the processor and is not specifically limited in this application.

[0133] S2227B: The multiple first processors decode the to-be-updated sets within the group of the multiple first processors according to the masks of the to-be-updated sets within the group corresponding to the multiple first processors respectively, and obtain the to-be-updated parts of the target parameters corresponding to the multiple processors respectively.

[0134] In simple terms, each first processor first obtains the Mask of the set to be updated in the group, and then multiple first processors transmit the Mask of the set to be updated in the group. Then, multiple first processors transmit the set to be updated in the group. In this way, each first processor can obtain the to-be-updated parts of the target parameters corresponding to each of the multiple processors after decoding using the MASK based on the obtained set to be updated in the group of all processors and the Mask of the set to be updated in the group.

[0135] It should be understood that since the positions of the to-be-updated sets within the group of multiple first processors in the target parameters (tensor) in the processor are different and the values ​​are also different, after data transmission, each first processor needs to decode the MASK of the to-be-updated sets within the group of all processors received and the to-be-updated sets within the group, determine the position of the to-be-updated sets within the group in the tensor, and thus obtain the to-be-updated part of the target parameter.

[0136] In a specific implementation, the transmission of the Mask of the to-be-updated set within the group between the above-mentioned multiple first processors can be implemented by the Allreduce algorithm in the aforementioned content. For details, please refer to the aforementioned content, which will not be repeated here.

[0137] In a specific implementation, the transmission of the to-be-updated set within the group by the above-mentioned multiple first processors can be realized by the Allgather algorithm. Specifically, the transmission of the to-be-updated set within the group between the first processors of multiple small groups so that each first processor of the multiple small groups obtains the to-be-updated set within the group of the first processors of multiple small groups, including: each first processor of the multiple first processors puts the to-be-updated set within the group into the cache of the processor; each first processor of the multiple first processors transmits the to-be-updated set within the group in the cache with other first processors, and puts the received to-be-updated set within the group into the cache of the processor, so that each first processor obtains the to-be-updated set within the group of the multiple first processors.

[0138] In one embodiment, referring to the embodiments of Tables 1 to 3, it can be known that the data lengths in the GPUs participating in Allgather must be the same, otherwise some GPUs may not be able to exchange all the data during the cache block exchange and overwriting process. Since the amount of data in the set to be updated in each first processor may be different, some first processors may be filled with data during data transmission so that the amount of data used for data communication in each first processor is the same. Through the Allgather algorithm, after each first processor obtains the set to be updated in the group of multiple first processors, it can delete the filled data first, and then decode the set to be updated in the group according to the Mask of the set to be updated in the group to obtain the part to be updated of the target parameter of each GPU.

[0139] The above method is used for inter-group transmission. During the entire transmission process, only MASK and the set to be updated within the group are transmitted. Compared with other methods (transmitting complete tensors for data), the communication volume is greatly reduced, thereby improving communication efficiency and increasing the training speed of the NAS system.

[0140] In summary, this application provides three methods to achieve communication between multiple processors in a NAS system:

[0141] The first method is to use the Mask union method. For details, please refer to the previous Figure 4 Steps S221A to S224A of the embodiment and optional steps thereof;

[0142] The second method is to communicate within the group first and then between groups, where inter-group communication is achieved through Mask union. For details, please refer to Figure 6 Steps S221B to S222B, steps S2221B to S2223B and optional steps thereof of the embodiment;

[0143] The third method is to communicate within the group first and then between groups, where the inter-group communication is achieved by directly transmitting the part to be updated. For details, please refer to Figure 6 Steps S221B to S222B of the embodiment, Figure 7 Steps S2224B to S2227B of the embodiment and optional steps thereof.

[0144] It should be understood that in different application scenarios, the communication volume required by the three methods is different. In some application scenarios, the communication volume of the first method is the lowest, in some scenarios, the communication volume of the second method is the lowest, and in some scenarios, the communication volume of the third method is the lowest. Therefore, before communicating, the communication volume required by the three methods can be calculated first, and then the method with the lowest communication volume can be selected for communication between multiple processors, thereby maximizing the training efficiency of the NAS system.

[0145] In summary, the data processing method provided by the present application only transmits the to-be-updated part of the target parameter of each processor during communication between multiple processors. Compared with other methods in which all data on the processor is transmitted, this method can greatly reduce the communication volume and improve the training efficiency of the NAS system.

[0146] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, correspondingly, related equipment for cooperating in implementing the above scheme is also provided below.

[0147] Figure 8 The present application provides a data processing device 800, which is applied to a neural network structure search NAS system. The system can be Figure 1 The system described in the embodiment includes a plurality of processors, each processor is used to train a sub-network in the search super network, and the device includes a training unit 810, a transmission unit 820 and an encoding unit 830, wherein:

[0148] The training unit 810 is used to perform current iteration training on the corresponding sub-network through each processor of the multiple processors to obtain the to-be-updated part of the target parameter of each processor;

[0149] The transmission unit 820 is used to transmit the information of the to-be-updated part of the target parameter obtained between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors.

[0150] In one embodiment, the data processing device 800 further includes an encoding unit 830, which is used to encode the part to be updated of the target parameter through each processor to obtain a mask of the part to be updated; the information of the part to be updated includes: the part to be updated and the mask of the part to be updated.

[0151] In one embodiment, the transmission unit 820 is used to transmit the mask of the part to be updated between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the mask of the part to be updated corresponding to each of the multiple processors; the transmission unit 820 is used to obtain the union of the masks of the parts to be updated corresponding to each of the multiple processors through each of the multiple processors; the transmission unit 820 is used to determine the overall part to be updated corresponding to each of the multiple processors according to the union of the masks through each of the multiple processors; the transmission unit 820 is used to transmit the determined overall part to be updated corresponding to each of the multiple processors between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the part to be updated of the target parameter corresponding to each of the multiple processors.

[0152] In one embodiment, the transmission unit 820 is used to place the mask of the corresponding part to be updated into the cache of each processor through each of the multiple processors; the transmission unit 820 is used to transmit the mask in the cache between each of the multiple processors and at least one other processor, and place the received mask into the cache of the processor, so that each of the multiple processors obtains the mask of the corresponding part to be updated of the multiple processors.

[0153] In one embodiment, the transmission unit 820 is used to place the corresponding overall part to be updated determined by the transmission into the cache of each processor through each of the multiple processors; the transmission unit 820 is used to transmit the overall part to be updated in the cache between each of the multiple processors and at least one other processor, and update the part to be updated in the cache based on the received part to be updated.

[0154] In one embodiment, multiple processors are divided into multiple groups, and each of the multiple groups includes at least two processors with the same corresponding sub-network structure; the transmission unit 820 is used to transmit the to-be-updated parts of the target parameters between at least two processors in the same group, so that the first processor in the same group obtains the to-be-updated set in the group, wherein the to-be-updated set in the group includes the to-be-updated parts of the target parameters corresponding to all processors in the group; the transmission unit 820 is used to transmit the to-be-updated set in the group between the first processors included in the multiple groups, so that each first processor of the multiple first processors obtains the to-be-updated parts of the target parameters corresponding to the multiple processors; the transmission unit 820 is used to multicast the to-be-updated parts of the target parameters corresponding to the multiple processors through each first processor of the multiple first processors, so that each processor in the same group obtains the to-be-updated parts of the target parameters corresponding to the multiple processors.

[0155] In one embodiment, the transmission unit 820 is used to place the to-be-updated part of the target parameter into the cache of the processor through each of at least two processors in the same group; the transmission unit 820 is used to transmit the to-be-updated part in the cache between each of at least two processors in the same group and other processors, and update the to-be-updated part in the cache of the processor based on the received to-be-updated part, so that the first processor in the same group obtains the to-be-updated set within the group.

[0156] In one embodiment, the encoding unit 830 is used to encode the set to be updated in the group through each first processor to obtain the mask of the set to be updated in the group; the transmission unit 820 is used to transmit the mask of the set to be updated in the group between multiple first processors included in multiple small groups, so that each of the multiple first processors obtains the mask of the set to be updated in the group corresponding to each of the multiple first processors; the transmission unit 820 is used to obtain the to-be-updated parts of the target parameters corresponding to each of the multiple processors based on the masks of the set to be updated in the group corresponding to each of the multiple first processors through each first processor.

[0157] In one embodiment, the transmission unit 820 is used to obtain the union of masks of the sets to be updated in the group corresponding to each of the multiple first processors based on the masks of the sets to be updated in the group corresponding to each of the multiple first processors through the first processor; the transmission unit 820 is used to determine the overall set to be updated in the group corresponding to each of the multiple first processors according to the union of the masks of the sets to be updated in the group through the first processor; the transmission unit 820 is used to transmit the determined overall set to be updated in the group between the multiple first processors, so that each first processor obtains the to-be-updated part of the target parameters corresponding to each of the multiple processors.

[0158] In one embodiment, the transmission unit 820 is used to transmit the intra-group set to be updated between multiple first processors, so that each first processor obtains the intra-group set to be updated of multiple first processors; the transmission unit 820 is used to decode the intra-group set to be updated of multiple first processors according to the masks of the intra-group set to be updated corresponding to each of the multiple first processors through the multiple first processors, and obtain the to-be-updated parts of the target parameters corresponding to each of the multiple processors.

[0159] In one embodiment, the processor is a GPU.

[0160] In summary, the data processing device provided by the present application, when calling multiple processors for gradient update in the NAS system, only transmits the to-be-updated part of the target parameter of each processor. Compared with other methods in which all parameters on the processor are transmitted, this method can greatly reduce the communication volume and improve the training efficiency of the NAS system.

[0161] Fig. 9 Schematic diagram of the structure of an electronic device 900 provided in an embodiment of the present application. The electronic device 900 may be the processor in the above content. Fig. 9 As shown, the electronic device 900 includes: a processing unit 910, a communication interface 920, a memory 930, and a bus 940. The processing unit 910, the communication interface 920, and the memory 930 can be interconnected through an internal bus 940, and can also communicate through other means such as wireless transmission. The embodiment of the present application takes the connection through the bus 940 as an example. The bus 940 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 940 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0162] The processing unit 910 may include one or more hardware chips, which may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or a combination thereof. The PLDs may be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. The processing unit 910 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 930, which enables the electronic device 900 to provide a wide variety of services.

[0163] The memory 930 is used to store program codes, and is controlled by the processing unit 910 to execute the above Figure 2-Figure 7 The processing steps of the processor in any embodiment of the present invention. The program code may include one or more software modules. The one or more software modules may be Figure 8 The software modules provided in the illustrated embodiment, such as a training unit, a transmission unit and an encoding unit, wherein the training unit is used to perform current iterative training on the corresponding sub-network through each of the multiple processors to obtain the part of the target parameter to be updated of each processor; the transmission unit is used to transmit the information of the part of the target parameter to be updated obtained between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the part of the target parameter to be updated corresponding to each of the multiple processors; the encoding unit is used to encode the part of the target parameter to be updated through each processor to obtain a mask of the part to be updated, which can be specifically used to execute steps S210-step S220 and optional steps, steps S221A-step S224A and optional steps, steps S221B-step S222B and optional steps, steps S2221B-step S2223AB and optional steps, steps S2224B-step S2227B and optional steps of the aforementioned method, and can also be used to execute Figure 2-Figure 7 Other steps executed by the processor described in the embodiment will not be described in detail here.

[0164] The memory 930 may include a volatile memory (Volatile Memory), such as a random access memory (Random Access Memory, RAM); the memory 930 may also include a non-volatile memory (Non-Volatile Memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (Flash Memory), a hard disk drive (Hard Disk Drive, HDD) or a solid-state drive (SSD); the memory 930 may also include a combination of the above types.

[0165] The communication interface 920 can be a wired interface (such as an Ethernet interface), an internal interface (such as a high-speed serial computer expansion bus (Peripheral Component Interconnect express, PCIe) bus interface), a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or a wireless local area network interface) for communicating with other devices or modules.

[0166] Need to explain, Fig. 9 This is only a possible implementation of the embodiment of the present application. In practical applications, the electronic device may also include more or fewer components, which is not limited here. Figure 2-Figure 7 The relevant explanations in the embodiments are not repeated here.

[0167] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a processor, Figure 2-Figure 7 The method flow shown is realized.

[0168] The present application also provides a computer program product. When the computer program product is run on a processor, Figure 2-Figure 7 The method flow shown is realized.

[0169] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD), or a semiconductor medium. The semiconductor medium may be an SSD.

[0170] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: Applied in a neural network structure search NAS system, the system includes a plurality of processors, each processor is used to train a sub-network in a search super network, each processor corresponds to a target parameter and a sub-network, and the method includes: Each of the plurality of processors performs current iterative training on the corresponding target parameter and sub-network to obtain a to-be-updated portion of the target parameter of each processor; Each of the plurality of processors performs encoding processing on the part to be updated to obtain a mask of the part to be updated; Each of the multiple processors transmits information of the part to be updated to at least one other processor, so that each of the multiple processors obtains the part to be updated of the target parameter corresponding to each of the multiple processors, and the information of the part to be updated includes: a mask of the part to be updated.

2. The method according to claim 1, characterized in that The information of the to-be-updated part of the target parameter transmitted between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors, includes: Each of the multiple processors transmits the mask of the to-be-updated portion to at least one other processor, so that each of the multiple processors obtains the mask of the to-be-updated portion corresponding to each of the multiple processors; Each of the multiple processors obtains a union of masks of the to-be-updated parts corresponding to each of the multiple processors; Each of the plurality of processors determines, according to the union of the masks, the overall to-be-updated parts corresponding to the plurality of processors; Each of the multiple processors transmits the determined corresponding overall to-be-updated parts to at least one other processor, so that each of the multiple processors obtains the to-be-updated parts of the target parameters corresponding to the multiple processors.

3. The method according to claim 2, characterized in that Transmitting the mask of the to-be-updated portion between each of the multiple processors and at least one other processor so that each of the multiple processors obtains the mask of the to-be-updated portion corresponding to each of the multiple processors includes: Each of the plurality of processors places the mask of the to-be-updated portion corresponding to the processor into a cache of the processor; Each of the multiple processors transmits the mask in the cache to at least one other processor, and puts the received mask into the cache of the processor, so that each of the multiple processors obtains the mask of the to-be-updated part corresponding to each of the multiple processors.

4. The method according to claim 3, characterized in that Each of the plurality of processors transmits the determined corresponding overall to-be-updated parts to at least one other processor, so that each of the plurality of processors obtains the to-be-updated parts of the target parameters corresponding to the plurality of processors, including: Each of the plurality of processors places the respective corresponding overall to-be-updated portion determined by the transmission into a cache of each processor; Each processor of the plurality of processors transmits the overall to-be-updated portion in the cache to at least one other processor, and updates the to-be-updated portion in the cache based on the received to-be-updated portion.

5. The method according to claim 1, characterized in that The plurality of processors are divided into a plurality of groups, each of the plurality of groups comprising at least two processors having the same corresponding sub-network structure; Each of the multiple processors transmits information about the to-be-updated portion of the target parameter to at least one other processor, so that each of the multiple processors obtains the to-be-updated portion of the target parameter corresponding to each of the multiple processors, including: At least two processors in the same group transmit the to-be-updated parts of the target parameters respectively, so that the first processor in the same group obtains the to-be-updated set in the group, wherein the to-be-updated set in the group includes the to-be-updated parts of the target parameters corresponding to all processors in the group; The first processors respectively included in the plurality of groups transmit the to-be-updated set within the group, so that each of the plurality of first processors obtains the to-be-updated part of the target parameter corresponding to each of the plurality of processors; Each of the plurality of first processors multicasts the to-be-updated portion of the target parameter corresponding to each of the plurality of processors, so that each processor in the same group obtains the to-be-updated portion of the target parameter corresponding to each of the plurality of processors.

6. The method according to claim 5, characterized in that The at least two processors in the same group transmit the to-be-updated part of the target parameter respectively to each other, so that the first processor in the same group obtains the to-be-updated set in the group, including: Each of the at least two processors in the same group places the to-be-updated portion of the target parameter into a cache of the processor; Each of the at least two processors in the same group transmits the part to be updated in the cache to other processors, and updates the part to be updated in the cache of the processor based on the received part to be updated, so that the first processor in the same group obtains the set to be updated in the group.

7. The method according to claim 6, characterized in that The first processors of the plurality of groups transmit the to-be-updated set within the group so that each first processor of the plurality of groups obtains the to-be-updated part of the target parameter corresponding to each of the plurality of processors, including: Each of the first processors performs encoding processing on the set to be updated in the group to obtain a mask of the set to be updated in the group; The plurality of first processors included in the plurality of small groups transmit the mask of the set to be updated in the group between each other, so that each of the plurality of first processors obtains the mask of the set to be updated in the group corresponding to each of the plurality of first processors; Each first processor obtains the to-be-updated parts of the target parameters corresponding to each of the plurality of first processors based on the masks of the to-be-updated sets within the group corresponding to each of the plurality of first processors.

8. The method according to claim 7, characterized in that Each of the first processors obtains the to-be-updated parts of the target parameters corresponding to each of the plurality of first processors based on the masks of the to-be-updated sets in the group corresponding to each of the plurality of first processors, including: The first processor obtains, based on the masks of the to-be-updated sets in the group corresponding to the multiple first processors respectively, a union of the masks of the to-be-updated sets in the group corresponding to the multiple first processors respectively; The first processor determines, according to a union of the masks of the sets to be updated in the group, the overall sets to be updated in the group corresponding to each of the plurality of first processors; The determined set to be updated within the overall group is transmitted between the multiple first processors, so that each first processor obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors.

9. The method according to claim 7, characterized in that: Each of the first processors obtains the to-be-updated parts of the target parameters corresponding to each of the plurality of first processors based on the masks of the to-be-updated sets in the group corresponding to each of the plurality of first processors, including: The plurality of first processors transmit the to-be-updated set within the group among themselves, so that each first processor obtains the to-be-updated set within the group of the plurality of first processors; The multiple first processors decode the to-be-updated sets within the group of the multiple first processors according to the masks of the to-be-updated sets within the group corresponding to the multiple first processors respectively, and obtain the to-be-updated parts of the target parameters corresponding to the multiple processors respectively.

10. The method according to any one of claims 1 to 9, characterized in that: The processor is a GPU.

11. A data processing device, characterized in that: Applied to a neural network structure search NAS system, the system includes a plurality of processors, each processor is used to train a sub-network in a search super network, each processor corresponds to a target parameter and a sub-network, and the device includes: A training unit, the training unit is used to perform current iterative training on the corresponding target parameters and sub-networks through each of the multiple processors to obtain a to-be-updated portion of the target parameters of each processor; an encoding unit, the encoding unit being configured to perform encoding processing on the to-be-updated portion of the target parameter through each of the plurality of processors to obtain a mask of the to-be-updated portion; A transmission unit, wherein the transmission unit is used to transmit the information of the to-be-updated part of the target parameter obtained between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the to-be-updated part of the target parameter corresponding to each of the multiple processors, and the information of the to-be-updated part includes: a mask of the to-be-updated part.

12. The device according to claim 11, characterized in that The transmission unit is used to transmit the mask of the to-be-updated part between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the mask of the to-be-updated part corresponding to each of the multiple processors; The transmission unit is used to obtain, through each of the multiple processors, a union of masks of the to-be-updated parts corresponding to each of the multiple processors; The transmission unit is used to determine the overall to-be-updated parts corresponding to each of the multiple processors according to the union of the masks through each of the multiple processors; The transmission unit is used to transmit the determined corresponding overall parts to be updated between each of the multiple processors and at least one other processor, so that each of the multiple processors obtains the parts to be updated of the target parameters corresponding to the multiple processors.

13. The device according to claim 12, characterized in that The transmission unit is used to place the corresponding overall to-be-updated parts determined by the transmission into the cache of each processor through each of the multiple processors; The transmission unit is used to transmit the overall to-be-updated portion in the cache between each of the multiple processors and at least one other processor, and to update the to-be-updated portion in the cache based on the received to-be-updated portion.

14. The device according to claim 11, characterized in that The plurality of processors are divided into a plurality of groups, each of the plurality of groups comprising at least two processors having the same corresponding sub-network structure; The transmission unit is used to transmit the to-be-updated part of the target parameter between at least two processors in the same group, respectively, so that the first processor in the same group obtains the to-be-updated set in the group, wherein the to-be-updated set in the group includes the to-be-updated parts of the target parameter corresponding to all processors in the group; The transmission unit is used to transmit the to-be-updated set within the group through the first processors respectively included in the plurality of groups, so that each of the plurality of first processors obtains the to-be-updated part of the target parameter corresponding to each of the plurality of processors; The transmission unit is used to multicast the to-be-updated parts of the target parameters corresponding to each of the multiple first processors through each of the multiple first processors, so that each processor in the same group obtains the to-be-updated parts of the target parameters corresponding to each of the multiple processors.

15. The device according to claim 14, characterized in that The transmission unit is used to put the to-be-updated part of the target parameter into the cache of the processor through each of the at least two processors in the same group; The transmission unit is used to transmit the part to be updated in the cache between each of at least two processors in the same group and other processors, and update the part to be updated in the cache of the processor based on the received part to be updated, so that the first processor in the same group obtains the set to be updated in the group.

16. The device according to claim 15, characterized in that The encoding unit is used to perform encoding processing on the set to be updated in the group through each of the first processors to obtain a mask of the set to be updated in the group; The transmission unit is used to transmit the mask of the set to be updated in the group between the multiple first processors included in the multiple groups, so that each of the multiple first processors obtains the mask of the set to be updated in the group corresponding to each of the multiple first processors; The transmission unit is used to obtain, through each first processor, the to-be-updated parts of the target parameters corresponding to each of the multiple processors based on the masks of the to-be-updated sets within the group corresponding to each of the multiple first processors.

17. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computing device, cause the computing device to perform the method according to any one of claims 1 to 10.

18. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor executes the code in the memory to perform the method according to any one of claims 1 to 10.

19. A neural network search system, characterized in that: The system comprises a plurality of processors, and the plurality of processors execute the method according to any one of claims 1 to 10.

Images