Data processing method and device, electronic equipment and computer readable storage medium
By discarding the output data in network model training and transmitting only data that is not set to zero, the problem of resource requirements optimization in network model training is solved, and resource consumption is reduced and training performance is improved.
Patent Information
- Application Number
- CN202411203829.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
AI Technical Summary
During the training process of network model, as the model scale increases, the demand for computing resources, communication resources, etc. increases. The existing technology still needs to transmit complete dropout data results when transmitting between different layers, resulting in waste of hardware resources and Resource consumption increases, making it difficult to optimize resource requirements.
During the training process of the network model, the output data of the first target layer is discarded, and only the data that is not set to 0 is transmitted to the second target layer, and the second target layer is passed through the second target layer to be transmitted in the first output data according to the data to be transmitted. Process the corresponding location to reduce the consumption of transmission and storage resources.
By only transmitting data that is not set to zero, the amount of transmitted data and resource usage are reduced, the efficiency and accuracy of network model training are improved, resource requirements are optimized, and training performance and reliability are improved.
Smart Images

Figure CN119067190A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method and device, an electronic device, and a computer-readable storage medium. Background Art
[0002] As the scale of large network models increases, the parameters they contain are also increasing, and the computing resources and communication resources required for network model training are also increasing. Therefore, in the process of network model training, it is very important to optimize resource requirements. Summary of the invention
[0003] The present disclosure provides a data processing method and device, an electronic device, and a computer-readable storage medium.
[0004] In a first aspect, the present disclosure provides a data processing method, the data processing method comprising:
[0005] During the training process of the network model, a discard operation is performed on the first output data of the first target layer of the network model to obtain data to be transmitted, wherein the discard operation represents setting the data values of some positions in the first output data to 0, and the data to be transmitted represents the data in the first output data that are not set to 0;
[0006] Sending the data to be transmitted to a second target layer of the network model;
[0007] The data to be transmitted is processed through the second target layer according to the data to be transmitted and a corresponding position of the data to be transmitted in the first output data.
[0008] In a second aspect, the present disclosure provides a data processing device, the data processing device comprising:
[0009] A first processing module is used to discard the first output data of the first target layer of the network model during the training process of the network model to obtain data to be transmitted, wherein the discard operation represents setting the data values of some positions in the first output data to 0, and the data to be transmitted represents the data in the first output data that are not set to 0;
[0010] A transmission module, used for sending the data to be transmitted to the second target layer of the network model;
[0011] The second processing module is used to process the data to be transmitted through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data.
[0012] In a third aspect, the present disclosure provides an electronic device, comprising: multiple processing cores; and an on-chip network configured to exchange data between the multiple processing cores and external data; wherein one or more instructions are stored in one or more of the processing cores, and one or more of the instructions are executed by one or more of the processing cores, so that one or more of the processing cores can execute the above-mentioned data processing method.
[0013] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method when executed by a processor.
[0014] In the embodiments provided by the present disclosure, during the training process of the network model, the first output data of the first target layer of the network model can be discarded to obtain only the data that is not set to 0 in the first output data as the data to be transmitted, and then the data to be transmitted can be sent to the second target layer of the network model. In this way, only the data that is not set to 0 can be transmitted, and there is no need to transmit the complete data containing the data set to 0 and the data not set to 0 after the discarding operation, which reduces the amount of transmitted data, reduces the consumption and occupation of hardware resources, optimizes resource requirements, and improves the training performance of the network model. After the data to be transmitted is sent to the second target layer, the second target layer can process the data to be transmitted in combination with the corresponding position of the data to be transmitted in the first output data. In this way, the second target layer can accurately process the data to be transmitted, improve efficiency and accuracy, and thus ensure the reliability and accuracy of network model training.
[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:
[0017] Figure 1 Schematic diagram of the neuron processing principle in the dropout operation in the embodiment of the present disclosure;
[0018] Figure 2 Schematic diagram of data transmission principle in dropout operation in an embodiment of the present disclosure;
[0019] Figure 3A flowchart of a data processing method provided by an embodiment of the present disclosure;
[0020] Figure 4 A schematic diagram of a processing principle of a second target layer in an embodiment of the present disclosure;
[0021] Figure 5 This is a scene example diagram of a data processing method in an embodiment of the present disclosure;
[0022] Figure 6 This is an example diagram of a random seed determination scenario in an embodiment of the present disclosure;
[0023] Figure 7 A block diagram of a data processing device provided in an embodiment of the present disclosure;
[0024] Figure 8 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0026] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.
[0027] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0028] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.
[0029] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.
[0030] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are briefly introduced below.
[0031] Dropout: Dropout is a regularization technique used in the network model training process. It reduces the risk of overfitting by randomly setting the output of some neurons to zero during the training process. It enables different parts of the network model to learn more robust and independent feature representations, thereby improving the generalization ability of the network model and effectively preventing the network model from overfitting the training data.
[0032] See also Figure 1 As shown in FIG. 1 , it is a schematic diagram of the neuron processing principle in the dropout operation in the embodiment of the present disclosure. During each forward inference and reverse gradient propagation in the network model training process, some calculation results can be randomly set to 0 through the dropout operation. The random probability can be configured, such as Figure 1 Each circle in the figure represents a neuron, and the one with a cross in it represents a discarded neuron.
[0033] See also Figure 2 As shown, it is a schematic diagram of the data transmission principle in the dropout operation in the embodiment of the present disclosure, such as Figure 2 As shown in the figure, taking layer A and layer B in the network model as an example, the dropout operation can randomly set some values in the output of layer A to 0, and then the result after the dropout operation can be transmitted to layer B for use, or it can be stored in the forward process for use in reverse gradient propagation.
[0034] As the scale of network models increases, the resources required for network model training are also increasing, such as computing resources, storage resources, and communication resources. Therefore, in the process of network model training, it is very important to optimize resource requirements.
[0035] In the related art, a dropout method is provided for the training process of the network model, which can randomly set the output of a part of neurons to zero during the training process to reduce the risk of overfitting. However, when transmitting between different layers in the network model, it is still necessary to transmit the complete data results after dropout, that is, to transmit and store the data containing these zero-set data and non-zero-set data, resulting in a waste of hardware resources in the network model training process and an increase in the consumption of storage and transmission resources.
[0036] According to the data processing method of the embodiment of the present disclosure, during the training process of the network model, after the first output data of the first target layer is dropout-operated, only the data that is not set to 0 in the first output data is obtained as the data to be transmitted. In this way, only the data that is not set to 0 can be transmitted without transmitting the data set to 0, which reduces the amount of transmitted data and the occupancy of transmission resources and reduces the consumption of hardware resources. After being transmitted to the second target layer, the data to be transmitted can be processed according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data. In this way, when only the data that is not set to 0 is transmitted, the second target layer can also be processed accurately to ensure the accuracy of data processing and the reliability of network model training.
[0037] The data processing method according to the embodiment of the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the method can be executed by a server.
[0038] Figure 3 A flowchart of a data processing method provided by an embodiment of the present disclosure. Figure 3 , the method comprising:
[0039] S310: During the training process of the network model, a discard operation is performed on the first output data of the first target layer of the network model to obtain data to be transmitted, wherein the discard operation represents setting the data values at some positions in the first output data to 0, and the data to be transmitted represents the data in the first output data that are not set to 0.
[0040] In the embodiment of the present disclosure, the discarding operation can be characterized as a dropout operation, and the first target layer can be any network layer in the network model without limitation.
[0041] S320: Send the data to be transmitted to the second target layer of the network model.
[0042] The second target layer may be a network layer in the network model that has a data connection with the first target layer.
[0043] In the embodiments of the present disclosure, the first target layer and the second target layer can be deployed on different processing cores of the many-core system, or can be deployed on different computing units, or can be deployed on different devices, etc. For example, taking the deployment on different processing cores as an example, the processing core used to process the first target layer can perform the dropout operation and only extract the data that is not set to 0 as the data to be transmitted, and send the data to be transmitted to the processing core used to process the second target layer.
[0044] S330: Processing the data to be transmitted through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data.
[0045] In the embodiment of the present disclosure, since only data that are not set to 0 are transmitted, when the second target layer processes the data, it needs to know at which positions the data that are not set to 0 are, or at which positions the data are set to 0, so that the corresponding calculations can be performed accurately. Therefore, in the embodiment of the present disclosure, the data can be processed by the second target layer in combination with the corresponding positions of the data to be transmitted in the first output data, thereby obtaining the output data of the second target layer.
[0046] Specifically with respect to the method for determining the corresponding position of the data to be transmitted in the first output data, the present disclosure also provides several possible embodiments: 1) the corresponding position of the data to be transmitted in the first output data is determined based on a random number seed, wherein the random seed is used to implement a discard operation, and the random seed is transmitted to synchronize the first target layer and the second target layer, or the random seed is pre-stored in the storage space where the second target layer is located.
[0047] 2) The corresponding position of the data to be transmitted in the first output data is determined based on position information pre-stored in the storage space where the second target layer is located, and the position information includes position information that is set to 0 and / or not set to 0 in the first output data.
[0048] Furthermore, based on the same data processing method mentioned above, after discarding the output data of the second target layer, only the data that is not set to 0 is transmitted to other target layers connected to the second target layer. The other target layers can then process the data according to the corresponding position information and the data that is not set to 0.
[0049] It should be noted that in the embodiments of the present disclosure, there is no limitation on the network model, and the network model can be used to perform any of image processing tasks, voice processing tasks, text processing tasks, and video processing tasks. The output data or data to be transmitted of each layer in the network model includes any of images, voices, texts, and videos.
[0050] In the disclosed embodiment, during the training process of the network model, the first output data of the first target layer is discarded to obtain the data to be transmitted, so that after the discarding operation, only the data that is not set to 0 needs to be transmitted, thereby reducing the consumption of hardware resources, and then sending the data to be transmitted to the second target layer. Through the second target layer, the corresponding position of the data to be transmitted in the first output data can be determined according to the random seed used to implement the discarding operation, or the corresponding position information stored in the second target layer, and then the data to be transmitted is processed. In this way, the second target layer can accurately know which position data is set to 0 and not set to 0, so as to calculate and process the corresponding position data, improve the accuracy, and thus improve the accuracy and reliability of the network model training.
[0051] The data processing method in the embodiment of the present disclosure is described below.
[0052] In the embodiments of the present disclosure, when the dropout operation is implemented on hardware, the pseudo-random mechanism of the hardware can be used to perform processing based on a random seed. Specifically, in the embodiments of the present disclosure, several possible implementations of the dropout operation are provided, and different corresponding processing can be performed in the second target layer for different implementations of the dropout operation.
[0053] In the case of implementing dropout operation based on random position selection.
[0054] In some possible embodiments, in the above step S310, the first output data of the first target layer of the network model is discarded to obtain the data to be transmitted, including: based on the first random seed, generating a target selection position for the first output data; according to the target selection position, extracting the data at the corresponding position from the first output data to obtain the data to be transmitted.
[0055] That is, in the embodiment of the present disclosure, a random position can be generated, and data at the corresponding position can be extracted from the first output data. The extracted data is data that has not been set to 0, and the data that has not been extracted can be considered as discarded or set to 0.
[0056] In the case of the implementation method of the dropout operation, that is, when the dropout operation is implemented based on random position selection, in some possible embodiments, the above-mentioned step S330 processes the data to be transmitted through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data, including: determining the calculation position in the second target layer that needs to participate in the calculation according to the corresponding position of the data to be transmitted in the first output data, wherein the corresponding position of the data to be transmitted in the first output data is determined based on a second random seed, and the second random seed is related to the first random seed; through the second target layer, according to the calculation position in the second target layer that needs to participate in the calculation, the data to be transmitted is processed.
[0057] In the embodiment of the present disclosure, in order to ensure that the complete data after the dropout operation can be correctly restored in the second target layer, the second random seed of the second target layer needs to be related to the first random seed of the first target layer, specifically:
[0058] 1) In a possible embodiment, when a preset condition is met, a second random seed for a second target layer is determined according to first random seeds corresponding to each of a plurality of first target layers, wherein the preset condition includes that an input of the second target layer needs to come from a plurality of first target layers; or, the first target layer is a sublayer of the target layer, and an input of the second target layer needs to come from a plurality of sublayers of the target layer.
[0059] For example, a target layer is divided into multiple first target layers, and the multiple first target layers are deployed on different devices or different processing cores, and each corresponds to a first random seed, such as the first random seed 1 and the first random seed 2. The second target layer requires output data from multiple first target layers. In this case, the second random seed can be determined based on the first random seed 1 and the first random seed 2. For example, a random seed transformation function is used to transform and generate the second random seed based on the first random seed 1 and the first random seed 2, so that the calculation position that needs to participate in the calculation determined by the second random seed is consistent with the target selection position representing the entire target layer generated by the first random seed 1 and the first random seed 2.
[0060] In addition, in the embodiment of the present disclosure, the second target layer can also be divided into multiple sub-layers, and deployed in different devices or processing cores for calculation to further improve computing efficiency. In this case, similarly, the second random seed corresponding to each sub-layer of the second target layer can also be determined according to the first random seed corresponding to the first target layer from which its input needs to come.
[0061] 2) In a possible embodiment, when the input of the second target layer comes from only one first target layer, the second random seed is the same as the first random seed.
[0062] In this case, the embodiment of the present disclosure also includes: sending the first random seed to the second target layer; or, pre-storing the second random seed in the storage space where the second target layer is located.
[0063] Furthermore, in the embodiments of the present disclosure, for example, refer to Figure 4 As shown, it is a schematic diagram of the processing principle of a second target layer in an embodiment of the present disclosure, such as Figure 4 As shown, in combination with the corresponding hardware, the device or processing core where the second target layer is deployed may not restore the data that is set to 0, but may use the random generation method corresponding to the first target layer based on the second random seed to determine the location where the parallel computing unit in the second target layer needs to perform calculation for the received data to be transmitted, such as Figure 4 The black circles indicate where calculations need to occur, and the white circles indicate where calculations do not need to occur. Figure 4 The arrangement of the positions marked by the black circles and the white circles in is the same as the matrix size of the first output data and corresponds one to one, for example Figure 4 The first position in the figure is a black circle, which means that the data at this position is not set to 0 in the first output data and needs to be included in the calculation. In addition, it should be noted that Figure 4 For the convenience of description and explanation only, it is a schematic diagram describing the result of determining the positions where calculation needs to occur and the positions where calculation does not need to occur based on the second random seed, which can improve calculation efficiency.
[0064] In the case of implementing the dropout operation based on generating a mask matrix consisting of 0s and 1s.
[0065] In some possible embodiments, the first output data of the first target layer of the network model is discarded in the above step S310 to obtain data to be transmitted, including: based on a third random seed, generating a mask matrix with the same size as the first output data, wherein each position in the mask matrix takes a value of 0 or 1; obtaining second output data by multiplying the mask matrix and the first output data; and filtering out data that is not set to 0 from the second output data to obtain data to be transmitted.
[0066] For example, if the first output data is a 5*5 matrix, a 5*5 mask matrix containing only 0 and 1 can be randomly generated based on the third random seed, wherein the number of 0s in the mask matrix can be determined according to the set random probability. Then, the mask matrix and the first output data are multiplied, that is, the data at some positions in the first output data can be set to 0, so as to obtain the second output data containing the data set to 0. Then, the data that is not set to 0 can be filtered out in sequence from left to right and from top to bottom in the matrix to obtain the data to be transmitted. Of course, other orders can also be used to filter out in sequence to obtain the data to be transmitted, which is not limited in the embodiments of the present disclosure.
[0067] In the case of the implementation method of the dropout operation, that is, when the dropout operation is implemented based on generating a mask matrix including 0 and 1, in some possible embodiments, the above-mentioned step S330 processes the data to be transmitted through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data, including: converting the data to be transmitted into second output data according to the corresponding position of the data to be transmitted in the first output data, wherein the corresponding position of the data to be transmitted in the first output data is determined based on a fourth random seed, and the fourth random seed is related to the third random seed; and processing the second output data through the second target layer.
[0068] That is, in the embodiment of the present disclosure, 0 can be inserted into the corresponding random position of the received data to be transmitted based on the fourth random seed to restore the data to be transmitted to the second output data, that is, the complete data after the dropout operation is performed on the first output data of the first target layer, and the second output data is used as the input of the second target layer for processing.
[0069] In this case, in the embodiment of the present disclosure, the fourth random seed in the second target layer needs to be related to the third random seed in the first target layer to ensure accuracy, specifically:
[0070] 1) In a possible embodiment, when a preset condition is met, the fourth random seed of the second target layer is determined according to the third random seeds corresponding to each of the multiple first target layers, wherein the preset condition includes that the input of the second target layer needs to come from multiple first target layers; or, the first target layer is a sublayer of the target layer, and the input of the second target layer needs to come from multiple sublayers of the target layer.
[0071] 2) In a possible embodiment, when the input of the second target layer comes from only one of the first target layers, the third random seed is the same as the fourth random seed, and further includes: sending the third random seed to the second target layer; or, the fourth random seed is pre-stored in the storage space where the second target layer is located.
[0072] In the embodiments of the present disclosure, different methods can be used to implement a dropout operation on the first output data of the first target layer, and obtain data to be transmitted that only contains data that is not set to 0, so as to be sent to the second target layer, thereby reducing the amount of transmitted data. The second target layer can then correctly process the received data to be transmitted based on the corresponding random seed and method, thereby improving accuracy.
[0073] Further, in the embodiment of the present disclosure, in the forward reasoning process during the training process of the network model, only the data that is not set to 0 can be transmitted to the next network layer to reduce the transmission resource occupation of the hardware. In addition, in the embodiment of the present disclosure, after obtaining the data to be transmitted, the data to be transmitted can also be stored. Specifically, the data to be transmitted is stored in the storage space where the first target layer is located for reverse gradient propagation during the training process of the network model. For example, in the reverse gradient propagation process, the predicted output, i.e., the data to be transmitted and the actual label are used to calculate the loss function, and starting from the output layer, the gradient of the loss function with respect to the weight and bias is calculated. Among them, the first target layer can also store the random number seed corresponding to the data to be transmitted, so as to restore the position of the data to be transmitted in the original matrix based on the random number seed in the subsequent calculation; or, the first target layer stores the position information of the data to be transmitted in the original matrix, and the position information can be implemented in different forms, for example, it can be realized in the form of a 01 matrix, and the position of the data to be transmitted in the original matrix can be restored based on the position information in the subsequent calculation. The subsequent calculation, for example, can be a reverse gradient propagation calculation process. In this way, the user only needs to store the data that are not set to 0, without having to store the complete data including the data that are set to 0, which can also reduce the storage resource usage and save storage space.
[0074] In some embodiments of the present disclosure, the above data processing method can be applied to a many-core system, which includes multiple processing cores, the first target layer is deployed on the first processing core of the many-core system, and the second target layer is deployed on the second processing core of the many-core system.
[0075] In a possible embodiment, a discard operation is performed on the first output data of the first target layer of the network model to obtain data to be transmitted, and the data to be transmitted is sent to the second target layer of the model, including: a first processing core discards the first output data of the first target layer of the network model to obtain data to be transmitted, and sends the data to be transmitted to the second processing core.
[0076] The data to be transmitted is processed through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data, including: the second processing core receives the data to be transmitted sent by the first processing core, and processes the data to be transmitted through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data.
[0077] It should be noted that the specific implementation methods of the first processing core and the second processing core performing the above steps are the same as the implementation methods of the steps in the above data processing method, and will not be described in detail here.
[0078] In this way, in the embodiment of the present disclosure, when different processing cores execute calculations of each network layer in the network model, the data transmitted between the processing cores only includes the data to be transmitted that has not been set to 0, thereby reducing the consumption of transmission resources. Furthermore, when storage is required for reverse gradient propagation, the data to be transmitted can also be stored in the first processing core, and there is no need to store the complete data after dropout, thereby reducing the occupancy and consumption of storage resources, thereby optimizing the demand and occupancy of transmission resources and storage resources in the network model training process and improving performance.
[0079] The data processing method in the embodiment of the present disclosure is described below using a specific application scenario.
[0080] Taking the dropout operation as an example, the first target layer is deployed as the first processing core, and the second target layer is deployed as the second processing core. Figure 5 The figure shows a scenario example diagram of a data processing method in an embodiment of the present disclosure.
[0081] 1) If Figure 5 As shown, the first output data of the first target layer is [a11, a12, a13, a14, a15; a21, a22, a23, a24, a25; a31, a32, a33, a34, a35; a41, a42, a43, a44, a45; a51, a52, a53, a54, a55]. The first processing core randomly generates a 5*5 mask matrix based on the third random seed, multiplies the mask matrix and the first output data, and obtains the second output data. For example, the second output data is [a11, 0, a13,0,0; 0,a22,a23,0,a25; a31,0,a33,a34,0; a41,0,0,a44,0; 0,a52,a53,0,a55], and then filter out the data that is not set to 0 from the second output data to obtain the data to be transmitted. Taking the filtering from back to front as an example, the data to be transmitted is [a55,a53,a52,a44,a41,a34,a33,a31,a25,a23,a22,a13,a11], where Figure 5 Only some of the values of the data to be transmitted are shown.
[0082] 2) The first processing core sends the data to be transmitted to the second processing core.
[0083] In the case where the input of the second target layer comes from only one first target layer, the first processing core may also send the third random seed to the second target layer at the same time, and the fourth random seed of the second target layer is the same as the third random seed.
[0084] Alternatively, the same random seed may be stored in the first processing core and the second processing core during the deployment phase.
[0085] 3) The second processing core receives the data to be transmitted sent by the first processing core, and according to the fourth random seed, adopts the random generation method corresponding to the first processing core to convert the data to be transmitted into second transmission data. The second transmission data is used as the input of the second target layer. After being input into the second target layer, the second output data can be processed by the second target layer.
[0086] In this way, in the embodiment of the present disclosure, during the network model training process, when the first processing core transmits data to the second processing core, it only needs to transmit the data that is not set to 0, thereby reducing the demand and consumption of transmission resources.
[0087] Furthermore, in a possible scenario, if a target layer A and a target layer B are both divided into multiple sub-layers, and the number of sub-layers is different, and each sub-layer is deployed on a different processing core, then refer to Figure 6 The figure shows an example diagram of a random seed determination scenario in an embodiment of the present disclosure.
[0088] like Figure 6 As shown, the target layer A is divided into two sub-layers, layer A[1] and layer A[2], and layer A[1] is deployed on processing core 1, and layer A[2] is deployed on processing core 2, with random seeds α1 and α2 respectively. The target layer B is divided into three processing cores for execution, and the target layer B is divided into three sub-layers, layer B[1], layer B[2], and layer B[3]. Layer B[1] is deployed on processing core 3, layer B[2] is deployed on processing core 4, and layer B[3] is deployed on processing core 5, with random seeds β1, β2, and β3 respectively.
[0089] Layers B[1], B[2], and B[3] all need to use part of the output data of layers A[1] and A[2]. Therefore, in this case, both α1 and α2 need to be sent to processing cores 3, 4, and 5. Then, a random seed transformation function can be set to obtain β1, β2, and β3 based on α1 and α2. For example, β1=f1(α1,α2), β2=f2(α1,α2), and β3=f3(α1,α2).
[0090] Among them, f1, f2 and f3 can be different random seed transformation functions, and can make the data to be transmitted received by layer B[1], layer B[2] and layer B[3] respectively converted using the corresponding random seeds β1, β2 and β3 to obtain the overall input of the target layer B, and the output data of layer A[1] and layer A[2] are dropout-operated based on random seeds α1 and α2 respectively to obtain the overall output of the target layer A. The obtained overall input of the target layer B is consistent with the overall output of the target layer A. In this way, each sublayer of the target layer B can accurately convert the received data to be transmitted based on its own random seeds.
[0091] In this way, in the embodiment of the present disclosure, for the scenario of many-to-many data transmission between different target layers in the network model, the random seed of the sublayer receiving data can be determined based on the random seeds of each sublayer input, and the accurate random seed can be determined to accurately convert the complete data after the discard operation, thereby ensuring the accuracy of data processing.
[0092] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.
[0093] In addition, the present disclosure also provides a data processing device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any data processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.
[0094] Figure 7 A block diagram of a data processing device provided in an embodiment of the present disclosure.
[0095] Reference Figure 7 , an embodiment of the present disclosure provides a data processing device, the data processing device comprising:
[0096] The first processing module 71 is used to discard the first output data of the first target layer of the network model during the training process of the network model to obtain data to be transmitted, wherein the discard operation represents setting the data values of some positions in the first output data to 0, and the data to be transmitted represents the data in the first output data that are not set to 0;
[0097] A transmission module 72, configured to send the data to be transmitted to a second target layer of the network model;
[0098] The second processing module 73 is used to process the data to be transmitted through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data.
[0099] In some possible embodiments, when the first output data of the first target layer of the network model is discarded to obtain the data to be transmitted, the first processing module 71 is used to:
[0100] Based on a first random seed, generating a target selection position for the first output data;
[0101] A position is selected according to the target, and data at a corresponding position is extracted from the first output data to obtain data to be transmitted.
[0102] In some possible embodiments, when the data to be transmitted is processed through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data, the second processing module 73 is used to:
[0103] Determine, according to a corresponding position of the data to be transmitted in the first output data, a calculation position in the second target layer that needs to be calculated, wherein the corresponding position of the data to be transmitted in the first output data is determined based on a second random seed, and the second random seed is related to the first random seed;
[0104] The data to be transmitted is processed through the second target layer according to the calculation position in the second target layer that needs to participate in the calculation.
[0105] In some possible embodiments, the second processing module 73 is further configured to: determine the second random seed of the second target layer according to the first random seeds respectively corresponding to the plurality of first target layers when a preset condition is met;
[0106] The preset condition includes that the input of the second target layer needs to come from multiple first target layers; or, the first target layer is a sublayer of the target layer, and the input of the second target layer needs to come from multiple sublayers of the target layer.
[0107] In some possible embodiments, when the input of the second target layer comes from only one of the first target layers, the second random seed is the same as the first random seed, and the transmission module 72 is further used to: send the first random seed to the second target layer;
[0108] Alternatively, the second random seed is pre-stored in the storage space where the second target layer is located.
[0109] In some possible embodiments, when the first output data of the first target layer of the network model is discarded to obtain the data to be transmitted, the first processing module 71 is used to:
[0110] Based on a third random seed, generate a mask matrix having the same size as the first output data, wherein each position in the mask matrix has a value of 0 or 1;
[0111] Obtain second output data by multiplying the mask matrix and the first output data;
[0112] Data that is not set to 0 is filtered out from the second output data to obtain data to be transmitted.
[0113] In some possible embodiments, when the data to be transmitted is processed through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data, the second processing module 73 is used to:
[0114] converting the data to be transmitted into the second output data according to a corresponding position of the data to be transmitted in the first output data, wherein the corresponding position of the data to be transmitted in the first output data is determined based on a fourth random seed, and the fourth random seed is related to the third random seed;
[0115] The second output data is processed through the second target layer.
[0116] In some possible embodiments, the second processing module 73 is further configured to: determine the fourth random seed of the second target layer according to the third random seeds respectively corresponding to the plurality of first target layers when a preset condition is met;
[0117] The preset condition includes that the input of the second target layer needs to come from multiple first target layers; or, the first target layer is a sublayer of the target layer, and the input of the second target layer needs to come from multiple sublayers of the target layer.
[0118] In some possible embodiments, when the input of the second target layer comes from only one of the first target layers, the third random seed is the same as the fourth random seed, and the transmission module 72 is further used to: send the third random seed to the second target layer;
[0119] Alternatively, the fourth random seed is pre-stored in the storage space where the second target layer is located.
[0120] In some possible embodiments, the apparatus includes a many-core system, the many-core system includes a plurality of processing cores, the first target layer is deployed on a first processing core of the many-core system, and the second target layer is deployed on a second processing core of the many-core system;
[0121] When the first output data of the first target layer of the network model is discarded to obtain the data to be transmitted, and the data to be transmitted is sent to the second target layer of the model, the first processing core discards the first output data of the first target layer of the network model to obtain the data to be transmitted, and sends the data to be transmitted to the second processing core;
[0122] When the data to be transmitted is processed through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data, the second processing core receives the data to be transmitted sent by the first processing core, and processes the data to be transmitted through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data.
[0123] In some possible embodiments, the first target layer and the second target layer are respectively deployed in different devices, or respectively deployed in different processing cores;
[0124] The corresponding position of the data to be transmitted in the first output data is determined based on a random number seed, wherein the random seed is used to implement the discard operation, the random seed is transmitted to be synchronized in the first target layer and the second target layer, or the random seed is pre-stored in a storage space where the second target layer is located;
[0125] Alternatively, the corresponding position of the data to be transmitted in the first output data is determined based on position information pre-stored in the storage space where the second target layer is located, and the position information includes position information that is set to 0 and / or not set to 0 in the first output data.
[0126] In some possible embodiments, the first processing module 71 is further used to: store the data to be transmitted in the storage space where the first target layer is located, so as to be used for reverse gradient propagation in the training process of the network model.
[0127] In some possible embodiments, based on a multi-core system, the first processing module 71 may be located in the first processing core, the second processing module 73 may be located in the second processing core, and the first processing core 71 and the second processing core 73 may communicate and transmit data through the transmission module 72.
[0128] Figure 8A block diagram of an electronic device provided in an embodiment of the present disclosure.
[0129] Reference Figure 8 An embodiment of the present disclosure provides an electronic device, which includes multiple processing cores 801 and an on-chip network 802, wherein the multiple processing cores 801 are connected to the on-chip network 802, and the on-chip network 802 is used to exchange data between the multiple processing cores and external data.
[0130] One or more instructions are stored in one or more processing cores 801 , and the one or more instructions are executed by one or more processing cores 801 , so that the one or more processing cores 801 can execute the above-mentioned data processing method.
[0131] In some embodiments, the electronic device may be a brain-like chip, which can use vectorized computing and needs to load the weight information and other parameters of the neural network model through an external memory such as a double data rate (DDR) synchronous dynamic random access memory. Therefore, the batch processing used in the embodiments of the present disclosure has a higher computing efficiency.
[0132] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned data processing method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.
[0133] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.
[0134] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).
[0135] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.
[0136] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0137] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0138] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.
[0139] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0140] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0141] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0142] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0143] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.
Claims
1. A data processing method, characterized in that: include: During the training process of the network model, a discard operation is performed on the first output data of the first target layer of the network model to obtain data to be transmitted, wherein the discard operation represents setting the data values of some positions in the first output data to 0, and the data to be transmitted represents the data in the first output data that are not set to 0; Sending the data to be transmitted to a second target layer of the network model; The data to be transmitted is processed through the second target layer according to the data to be transmitted and a corresponding position of the data to be transmitted in the first output data.
2. The method according to claim 1, characterized in that The discarding operation on the first output data of the first target layer of the network model to obtain the data to be transmitted includes: Based on a first random seed, generating a target selection position for the first output data; A position is selected according to the target, and data at a corresponding position is extracted from the first output data to obtain data to be transmitted.
3. The method according to claim 2, characterized in that The processing of the data to be transmitted by the second target layer according to the data to be transmitted and a corresponding position of the data to be transmitted in the first output data includes: Determine, according to a corresponding position of the data to be transmitted in the first output data, a calculation position in the second target layer that needs to be calculated, wherein the corresponding position of the data to be transmitted in the first output data is determined based on a second random seed, and the second random seed is related to the first random seed; The data to be transmitted is processed through the second target layer according to the calculation position in the second target layer that needs to participate in the calculation.
4. The method according to claim 3, characterized in that The method further comprises: When a preset condition is met, determining the second random seed of the second target layer according to the first random seeds respectively corresponding to the plurality of first target layers; The preset condition includes that the input of the second target layer needs to come from multiple first target layers; or, the first target layer is a sublayer of the target layer, and the input of the second target layer needs to come from multiple sublayers of the target layer.
5. The method according to claim 3, characterized in that: In the case where the input of the second target layer comes from only one of the first target layers, the second random seed is the same as the first random seed, and the method further includes: sending the first random seed to the second target layer; Alternatively, the second random seed is pre-stored in the storage space where the second target layer is located.
6. The method according to claim 1, characterized in that The discarding operation on the first output data of the first target layer of the network model to obtain the data to be transmitted includes: Based on a third random seed, generate a mask matrix having the same size as the first output data, wherein each position in the mask matrix has a value of 0 or 1; Obtain second output data by multiplying the mask matrix and the first output data; Data that is not set to 0 is filtered out from the second output data to obtain data to be transmitted.
7. The method according to claim 6, characterized in that The processing of the data to be transmitted by the second target layer according to the data to be transmitted and a corresponding position of the data to be transmitted in the first output data includes: converting the data to be transmitted into the second output data according to a corresponding position of the data to be transmitted in the first output data, wherein the corresponding position of the data to be transmitted in the first output data is determined based on a fourth random seed, and the fourth random seed is related to the third random seed; The second output data is processed through the second target layer.
8. The method according to claim 7, characterized in that The method further comprises: When a preset condition is met, determining the fourth random seed of the second target layer according to the third random seeds corresponding to each of the plurality of first target layers; The preset condition includes that the input of the second target layer needs to come from multiple first target layers; or, the first target layer is a sublayer of the target layer, and the input of the second target layer needs to come from multiple sublayers of the target layer.
9. The method according to claim 7, characterized in that: In the case where the input of the second target layer comes from only one of the first target layers, the third random seed is the same as the fourth random seed, and the method further includes: Sending the third random seed to the second target layer; Alternatively, the fourth random seed is pre-stored in the storage space where the second target layer is located.
10. The method according to claim 1, characterized in that The first target layer and the second target layer are respectively deployed in different devices, or respectively deployed in different processing cores; The corresponding position of the data to be transmitted in the first output data is determined based on a random number seed, wherein the random seed is used to implement the discard operation, the random seed is transmitted to be synchronized in the first target layer and the second target layer, or the random seed is pre-stored in a storage space where the second target layer is located; Alternatively, the corresponding position of the data to be transmitted in the first output data is determined based on position information pre-stored in the storage space where the second target layer is located, and the position information includes position information that is set to 0 and / or not set to 0 in the first output data.
11. The method according to claim 1, characterized in that: The method further comprises: The data to be transmitted is stored in the storage space where the first target layer is located, so as to be used for reverse gradient propagation in the training process of the network model.
12. A data processing device, characterized in that: include: A first processing module is used to discard the first output data of the first target layer of the network model during the training process of the network model to obtain data to be transmitted, wherein the discard operation represents setting the data values of some positions in the first output data to 0, and the data to be transmitted represents the data in the first output data that are not set to 0; A transmission module, used for sending the data to be transmitted to the second target layer of the network model; The second processing module is used to process the data to be transmitted through the second target layer according to the data to be transmitted and the corresponding position of the data to be transmitted in the first output data.
13. An electronic device, characterized in that: include: Multiple processing cores; as well as An on-chip network is configured to exchange data between the multiple processing cores and external data; wherein one or more instructions are stored in one or more of the processing cores, and one or more of the instructions are executed by one or more of the processing cores, so that one or more of the processing cores can execute the data processing method described in any one of claims 1-11.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 11.
