A neural network computing method and related equipment

Abstract

This application discloses a neural network computation method and related apparatus. The method is executed by a neural network system including at least one convolutional layer. According to this method, after obtaining input data for a target convolutional layer, convolution computation is performed on the input data based on the weights of the target convolutional layer to obtain first output data. The first output data includes N first feature maps, where N is an integer greater than or equal to 1. Furthermore, based on the attention vector corresponding to the target convolutional layer, the first output data is computed to obtain second output data. The second output data includes N second feature maps that are robust to noise. The attention vector includes N elements, each of which represents the robustness of the corresponding first feature map to noise. Using this application embodiment can improve the accuracy of neural network computation.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a neural network calculation method and related equipment. Background Technology

[0002] In recent years, neural network (NN) computing has made groundbreaking progress, achieving high accuracy in many fields such as image recognition, speech recognition, and natural language processing. The core computation of neural networks is matrix-vector multiplication. Because matrix-vector multiplication is computationally intensive, neural networks require massive computing resources. Existing general-purpose processors are struggling to meet the computational demands of deep learning, making the design of dedicated chips an important development direction. Meanwhile, the emergence of circuit devices such as memristors, capacitor comparators, and voltage comparators has provided an efficient solution for neural network chip design. Among these, memristors, in addition to storing data, can also perform calculations, offering advantages such as high density, non-volatility, low power consumption, in-memory computation, and ease of 3D modeling, providing a highly efficient solution for the design of neural network systems.

[0003] While memristors can perform matrix-vector multiplication efficiently, in practical applications, due to inherent characteristics, they suffer from read / write errors, also known as device noise. This means that the set memristor conductance value does not precisely equal the required neural network weights, but rather falls within a distribution around the expected value. Improving the accuracy of neural networks and reducing the impact of noise from circuit components (such as memristors) on the accuracy of neural network calculations is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a neural network calculation method and related equipment to reduce the impact of noise from circuit devices in a neural network system on the accuracy of neural network calculations.

[0005] In a first aspect, embodiments of this application provide a computation method, which is executed by a neural network system including at least one convolutional layer, and the method may include:

[0006] The input data of the target convolutional layer is obtained, wherein the target convolutional layer is any one of the at least one convolutional layers; the input data is convolved based on the weights of the target convolutional layer to obtain first output data, wherein the first output data includes N first feature maps, where N is an integer greater than or equal to 1; the first output data is calculated based on the attention vector corresponding to the target convolutional layer to obtain second output data, wherein the second output data includes N second feature maps that are robust to noise, and the attention vector includes N elements, each of which represents the robustness of the corresponding first feature map to noise.

[0007] In existing neural network computation techniques, after receiving input data, the target convolutional layer performs convolution calculations on the input data based on its weights to obtain the output data. This output data is then directly used as the input data for the next layer. However, among all the feature maps (or feature data) in the output data of the target convolutional layer, the robustness of each feature map to noise varies; some feature maps are highly robust to noise, while others are poorly robust. Therefore, not all feature maps are suitable for direct input to the next layer. Therefore, this application proposes an attention vector to correct the output data obtained from weight-based convolution calculations. Specifically, in this embodiment, for any convolutional layer (such as a target convolutional layer) included in the neural network, in the process of obtaining the output data of the target convolutional layer, after performing convolution calculation on the input data based on the weights of the target convolutional layer to obtain the first output data, the first output data is further modified by the attention vector corresponding to the target convolutional layer. For example, feature maps with strong robustness to noise are strengthened, while feature maps with poor robustness to noise are weakened or even removed. The robustness of each feature map included in the extracted first output data to noise is comprehensively considered, and the best feature map is selected and input into the next layer for calculation of the next layer of the target convolutional layer, thereby reducing the impact of noise on the neural network system and improving the accuracy of neural network calculation.

[0008] In one possible implementation, the target convolutional layer includes M channels, and the method further includes: determining N channels from the M channels for performing the convolution calculation based on the attention vector, wherein the element values ​​of the elements included in the attention vector are greater than or equal to a preset threshold, and M is an integer greater than N.

[0009] In this embodiment, before performing convolution calculations on the input data based on the weights of the target convolutional layer to obtain the first output data, some channels of the target convolutional layer can be pre-closed based on the attention vector. Convolution calculations are then performed only on the remaining N channels to obtain feature maps with relatively strong noise robustness. Simultaneously, different feature maps are assigned corresponding weighting coefficients based on their varying noise robustness, thereby further refining the output feature maps with relatively strong noise robustness. This embodiment reduces the computational load of neural network inference by decreasing the number of channels, compressing the network model while maintaining computational accuracy. Furthermore, the attention vector can select the optimal feature data for the next layer of the target convolutional layer, preventing feature maps with poor noise robustness, such as those related to circuit devices, from flowing into the next layer. This reduces the impact of noise on the neural network system and improves the accuracy of neural network inference calculations.

[0010] In one possible implementation, the method further includes: adjusting the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system, wherein the adjustment process incorporates at least one of the simulated noise and actual noise of the circuit devices of the neural network system, and the trained network parameters include the weights of the target convolutional layer and the attention vector corresponding to the target convolutional layer.

[0011] Existing neural network training methods generally improve the robustness of each feature map obtained through convolution calculations using these weights by training the weights in the network parameters of the neural network system. Therefore, existing neural network training methods can be understood as a form of longitudinal training, where the robustness of each feature map to noise is improved by training the weights during feature map extraction. However, the robustness of each feature map extracted by the weights varies, and not all feature maps are suitable for subsequent computation. Therefore, this application proposes an attention vector to correct the output data obtained through weight-based convolution calculations. Specifically, in this application, for any convolutional layer (such as a target convolutional layer) in the neural network, after obtaining the first output data by performing convolution calculations on the input data based on the weights of the target convolutional layer, the first output data is further corrected using the attention vector corresponding to the target convolutional layer. This reduces the impact of noise on the neural network system and improves the accuracy of neural network calculations. Meanwhile, during training, the noise of the underlying circuit devices can be exposed to the software training layer and / or combined with the actual noise of the neural network system for hardware training. This can offset some of the accuracy impact caused by noise at the hardware level, thereby enabling the training of the neural network to adapt to the presence of noise. To a certain extent, this increases the robustness of the neural network weights and allows the attention vector to more accurately correct the output feature map.

[0012] In one possible implementation, adjusting the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system includes: inputting the training data into the neural network system to obtain third output data; calculating a loss value based on the third output data and the target output data; and updating the initial network parameters of the neural network according to the loss value.

[0013] In this embodiment, during the training of the network parameters of the neural network system, the network parameters are adjusted based on the training data to obtain network parameters that enable the neural network system to converge. Thus, in the process of obtaining the output data of the target convolutional layer, the input data is first convolved based on the weights of the target convolutional layer to obtain the first output data. Furthermore, the first output data is corrected by the attention vector corresponding to the target convolutional layer, thereby reducing the impact of noise on the neural network system and improving the accuracy of neural network calculation.

[0014] Secondly, embodiments of this application provide a neural network computing device, the device being applied to a neural network system including at least one convolutional layer, the device comprising:

[0015] An acquisition unit is configured to acquire input data of a target convolutional layer, wherein the target convolutional layer is any one of the at least one convolutional layers; a calculation unit is configured to perform convolution calculation on the input data based on the weights of the target convolutional layer to obtain first output data, wherein the first output data includes N first feature maps, where N is an integer greater than or equal to 1; the calculation unit is further configured to calculate on the first output data based on the attention vector corresponding to the target convolutional layer to obtain second output data, wherein the second output data includes N second feature maps that are robust to noise, and the attention vector includes N elements, each of which represents the robustness of the corresponding first feature map to noise.

[0016] In one possible implementation, the target convolutional layer includes M channels, and the apparatus further includes:

[0017] The determining unit is used to determine N channels for performing the convolution calculation from the M channels based on the attention vector, wherein the element values ​​of the elements included in the attention vector are greater than or equal to a preset threshold, and M is an integer greater than N.

[0018] In one possible implementation, the device further includes:

[0019] A training unit is used to adjust the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system. The adjustment process incorporates at least one of the simulated noise and actual noise of the circuit devices of the neural network system. The trained network parameters include the weights of the target convolutional layer and the attention vector corresponding to the target convolutional layer.

[0020] In one possible implementation, when adjusting the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system, the training unit is specifically used for:

[0021] The training data is input into the neural network system to obtain third output data; a loss value is calculated based on the third output data and the target output data; and the initial network parameters of the neural network are updated according to the loss value.

[0022] Thirdly, embodiments of this application provide a neural network training method. The weights and attention vectors in the neural network calculation method according to any one of the first aspects of this invention can be trained using the neural network training method described below. The method is performed by a neural network system including at least one convolutional layer, and the method may include:

[0023] The training data of the neural network system is obtained to obtain the third output data; the target convolutional layer is any one of the at least one convolutional layers, and the target convolutional layer is used to calculate the input data of the target convolutional layer through a first attention vector and L first weights to obtain the input data of the next layer of the target convolutional layer; wherein, the L first weights are used for convolution calculation to extract feature data in the input data of the target convolutional layer, the first attention vector is used to correct the feature data, and L is an integer greater than or equal to 1;

[0024] The third output data is compared with the target output data to calculate the first loss value;

[0025] The network parameters of the neural network system are updated based on the first loss value, and the network parameters include the first attention vector and the L first weights.

[0026] The training methods for existing neural network models generally improve the robustness of each feature data extracted by weights to noise by training the weights in the network parameters of the neural network system. Therefore, the existing neural network training methods can be understood as a kind of vertical training, that is, when extracting each feature, the robustness of each feature data to noise is improved by training the weights. However, the features extracted by weights also have different robustness to noise. Some feature data are highly robust to noise, while others are poorly robust. Therefore, not all features are suitable for entering the next layer of computation. Therefore, this application proposes an attention vector to modify each feature extracted by weights. Specifically, in this application embodiment, in any convolutional layer (such as a target convolutional layer) included in the neural network, the input data of the target convolutional layer is calculated by using a first attention vector and L first weights to obtain the output data of the target convolutional layer. The output data of the target convolutional layer is the input data of the next layer of the target convolutional layer. In the process of obtaining the input data for the next layer of the target convolutional layer, in addition to extracting feature data from the input data through L first weights, the extracted feature data is also modified through the first attention vector. For example, feature data that is robust to noise in the circuit devices of the neural network system is strengthened, while feature maps that are not robust to noise in the circuit devices of the neural network system are weakened or even removed. By comprehensively considering the robustness of each feature data to noise in the extracted feature data, the best feature data is selected and input into the next layer for the calculation of the next layer of the target convolutional layer, thereby reducing the impact of noise on the neural network system and improving the accuracy of the neural network.

[0027] In one possible implementation, the updated L first weights are L second weights, and the updated first attention vector is a second attention vector; the method further includes: inputting the training data into the neural network system to obtain fourth output data; the target convolutional layer calculates the input data of the target convolutional layer using the second attention vector and L third weights to obtain the input data of the next layer of the target convolutional layer; wherein, the L third weights correspond one-to-one with the L second weights, and each third weight is obtained by writing the corresponding second weight into the circuit device of the neural network system and combining it with the actual noise of the circuit device in the neural network system; comparing the fourth output data and the target output data to calculate a second loss value; and updating the second attention vector in the network parameters according to the second loss value.

[0028] In this embodiment, after updating the network parameters of the neural network system according to the first loss value, the network parameters of the neural network system are written into the actual circuit device. Specifically, L second weights can be written into the analog circuit device, and the second attention vector can be written into the digital circuit device (e.g., a digital storage module). At this time, due to the manufacturing process of the circuit device in the neural network system, when the weights are written into or read from the analog circuit device, the weights after writing or reading (i.e., the actual values) will deviate from the weights before writing or reading (i.e., the ideal values). That is, the L second weights written into the circuit device of the neural network system will be transformed into L third weights due to the influence of the circuit device noise during the operation of the circuit device in the neural network system. Therefore, in this embodiment, the network parameters of the neural network system are written into the actual circuit device, and the L second weights are fixed. The second attention vector is then retrained to obtain an updated second attention vector so that the final second attention vector can more accurately correct the extracted feature data.

[0029] In one possible implementation, the method further includes: setting elements in the initial attention vector that are less than or equal to the preset threshold to zero, while keeping elements in the vector that are greater than the preset threshold unchanged, to obtain the first attention vector.

[0030] In this embodiment, each element in the attention vector is used to evaluate the robustness of the corresponding feature data to noise. A larger element value indicates stronger robustness to noise and better noise resistance; a smaller element value indicates weaker robustness to noise and poorer noise resistance. Therefore, during neural network training, the initial attention vector is processed according to a preset threshold. Elements corresponding to feature maps with poor noise resistance (i.e., elements less than the preset threshold) are set to zero, while elements corresponding to feature maps with strong noise resistance (i.e., elements greater than the preset threshold) are retained. This results in a first attention vector composed of elements with some values ​​of zero and others with non-zero values. At this point, when the first attention vector corrects the feature data, by multiplying the first attention vector, which includes a portion of elements with zero values ​​and another portion of elements with non-zero values, with the input data of the target convolutional layer and L first weights, feature data that is robust to noise in the circuit devices of the neural network system can be selected, and feature data that is not robust to noise in the circuit devices of the neural network system can be directly removed. This prevents feature data that is not robust to noise in the circuit devices of the neural network system from flowing into the next layer of the target convolutional layer, thereby reducing the impact of noise in the circuit devices of the neural network system on the neural network system and improving the accuracy of the neural network.

[0031] In one possible implementation, the method further includes: processing the initial attention vector based on an activation function; for the processed initial attention vector, setting the elements in the vector that are less than or equal to the preset threshold to zero, while keeping the elements in the vector that are greater than the preset threshold unchanged, to obtain the first attention vector.

[0032] In this embodiment, during neural network training, the initial attention vector is processed using an activation function to obtain a processed initial attention vector whose element values ​​are distributed in a closed interval between 0 and 1. Then, this processed initial attention vector is further processed according to a preset threshold to obtain a first attention vector whose elements consist of a portion of zero values ​​and another portion of non-zero values ​​located in a closed interval between 0 and 1. At this point, on the one hand, since the non-zero elements of the first attention vector are located in a closed interval between 0 and 1, processing the attention vector using an activation function can improve the performance of the neural network, making it easier for the neural network to converge. On the other hand, when the first attention vector corrects the feature data, by multiplying the first attention vector (which includes a portion of zero values ​​and another portion of non-zero values) with the input data of the target convolutional layer and L first weights, feature data with strong noise robustness to the circuit devices in the neural network system can be selected, while feature data with poor noise robustness to the circuit devices in the neural network system can be directly removed. This prevents feature data with poor noise robustness to the circuit devices in the neural network system from flowing into the next layer of the target convolutional layer, thereby reducing the impact of noise from the circuit devices in the neural network system on the neural network system and improving the accuracy of the neural network.

[0033] In one possible implementation, setting elements in the vector less than or equal to the preset threshold to zero, while keeping elements in the vector greater than the preset threshold unchanged, to obtain the first attention vector includes: sorting the elements in the vector in descending order of value; determining the value of the Nth element after sorting as the preset threshold, where N is determined based on a preset pruning rate, the pruning rate being used to represent the ratio of the number of invalid elements in the vector to the total number of elements in the vector; setting elements in the vector less than or equal to the preset threshold to zero, while keeping elements in the vector greater than the preset threshold unchanged, to obtain the first attention vector.

[0034] In this embodiment, the values ​​of each element in the attention vector are used to evaluate the robustness of the corresponding feature data to noise. A larger element value indicates stronger robustness of the corresponding feature data to noise; a smaller element value indicates weaker robustness. Based on the values ​​of each element in the initial attention vector or the initial attention vector processed by the activation function, a preset threshold is determined. Then, the initial attention vector or the initial attention vector processed by the activation function is processed according to the preset threshold to obtain a first attention vector. When the first attention vector corrects the feature data, by multiplying the first attention vector (which includes some elements with zero values ​​and others with non-zero values) with the input data of the target convolutional layer and L first weights, feature data with strong noise robustness to the circuit devices in the neural network system can be selected, while feature data with poor noise robustness to the circuit devices in the neural network system can be directly removed. This prevents feature data with poor noise robustness to the circuit devices in the neural network system from flowing into the next layer of the target convolutional layer, thereby reducing the impact of noise from the circuit devices in the neural network system on the neural network system and improving the accuracy of the neural network.

[0035] In one possible implementation, the method further includes: combining the read / write errors of circuit devices in a simulated neural network system with L initial weights to obtain L first weights, wherein the L initial weights correspond one-to-one with the L first weights. Optionally, the read / write errors of circuit devices in the simulated neural network system follow a normal distribution, and the standard deviation of the normal distribution is related to the magnitude of the read / write errors of the circuit devices in the neural network system.

[0036] In this embodiment, during the training of the L first weights, simulated noise from the circuit components can be incorporated into the neural network training process. By exposing the noise from the circuit components in the underlying neural network system to the software training layer, the accuracy impact caused by hardware-level noise can be partially offset. This allows the neural network training to adapt to situations where noise exists, rather than simply representing precise weights. Therefore, this embodiment introduces noise into the network training process, which to some extent increases the robustness of the network weights.

[0037] In one possible implementation, updating the network parameters of the neural network system based on the first loss value includes: updating the initial attention vector and L initial weights based on the first loss value; updating the L first weights based on the updated L initial weights; and updating the first attention vector based on the updated initial attention vector.

[0038] In this embodiment, when updating the L first weights and the first attention vector, the initial attention vector and the L initial weights are updated via backpropagation based on the first loss value. For the updated initial attention vector (or the initial attention vector processed by the activation function), elements in the vector less than or equal to the preset threshold are set to zero, while elements in the vector greater than the preset threshold remain unchanged, thus updating the first attention vector. For the updated L initial weights, when performing the input data multiplication operation with the weights, noise is added to each element of the initial weights. Optionally, this noise can be a normally distributed noise, used to simulate the noise of circuit devices in a neural network system such as a memristor, thereby obtaining the updated L first weights.

[0039] Fourthly, embodiments of this application provide a neural network training apparatus applied to a neural network system including at least one convolutional layer, the apparatus comprising:

[0040] An input unit is used to acquire training data of the neural network system and obtain third output data; the target convolutional layer is any one of the at least one convolutional layers, and the target convolutional layer is used to calculate the input data of the target convolutional layer through a first attention vector and L first weights to obtain the input data of the next layer of the target convolutional layer; wherein, the L first weights are used for convolution calculation to extract feature data from the input data of the target convolutional layer, the first attention vector is used to correct the feature data, and L is an integer greater than or equal to 1;

[0041] The calculation unit is used to compare the third output data and the target output data to calculate the first loss value;

[0042] An update unit is used to update the network parameters of the neural network system based on the first loss value, the network parameters including the first attention vector and the L first weights.

[0043] In one possible implementation, the updated L first weights are L second weights, and the updated first attention vector is a second attention vector;

[0044] The input unit is further configured to input the training data into the neural network system to obtain second output data; the target convolutional layer calculates the input data of the target convolutional layer through the second attention vector and L third weights to obtain the input data of the next layer of the target convolutional layer; wherein, the L third weights correspond one-to-one with the L second weights, and each third weight is obtained by writing the corresponding second weight into the circuit device of the neural network system and combining it with the actual noise of the circuit device in the neural network system;

[0045] The calculation unit is further configured to compare the second output data and the target output data to calculate the second loss value;

[0046] The update unit is further configured to update the second attention vector in the network parameters according to the second loss value.

[0047] In one possible implementation, the device further includes:

[0048] The first processing unit is configured to, for the initial attention vector, set the elements in the vector that are less than or equal to the preset threshold to zero, and keep the elements in the vector that are greater than the preset threshold unchanged, to obtain the first attention vector.

[0049] In one possible implementation, the device further includes:

[0050] The first processing unit is used to process the initial attention vector based on the activation function; for the processed initial attention vector, the elements in the vector that are less than or equal to the preset threshold are set to zero, while the elements in the vector that are greater than the preset threshold are kept unchanged, to obtain the first attention vector.

[0051] In one possible implementation, the first processing unit is specifically used to: set elements in the vector less than or equal to the preset threshold to zero, and keep elements in the vector greater than the preset threshold unchanged to obtain the first attention vector;

[0052] Sort the elements in the vector in descending order of their numerical values;

[0053] The value of the Nth element after sorting is determined as the preset threshold, where N is determined based on a preset pruning rate, and the pruning rate is used to represent the ratio of the number of invalid elements in the vector to the total number of elements in the vector;

[0054] The elements in the vector that are less than or equal to the preset threshold are set to zero, while the elements in the vector that are greater than the preset threshold are kept unchanged, thus obtaining the first attention vector.

[0055] In one possible implementation, the device further includes:

[0056] The second processing unit is used to combine the read / write errors of the circuit devices in the simulated neural network system with the L initial weights to obtain the L first weights, wherein the L initial weights correspond one-to-one with the L first weights. Optionally, the read / write errors of the circuit devices in the simulated neural network system follow a normal distribution, and the standard deviation of the normal distribution is related to the magnitude of the read / write errors of the circuit devices in the neural network system.

[0057] In one possible implementation, the update unit is specifically used for:

[0058] Update the initial attention vector and L initial weights based on the first loss value;

[0059] Based on the updated L initial weights, the L first weights are updated;

[0060] The first attention vector is updated based on the updated initial attention vector.

[0061] Fifthly, embodiments of this application provide a neural network computing device, which includes a processor configured to support the neural network computing device in implementing corresponding functions in the neural network computing method provided in the first aspect. The neural network computing device may further include a memory coupled to the processor, which stores necessary program instructions and data for the neural network computing device. The neural network computing device may also include a communication interface for communicating with other devices or communication networks.

[0062] Sixthly, embodiments of this application provide a neural network training apparatus, which includes a processor configured to support the neural network computing device in implementing the corresponding functions in the neural network training method provided in the third aspect. The neural network training apparatus may further include a memory coupled to the processor, which stores necessary program instructions and data for the neural network training apparatus. The neural network training apparatus may also include a communication interface for communicating with other devices or communication networks.

[0063] In a seventh aspect, embodiments of this application provide a neural network processor configured to support the implementation of corresponding functions in the neural network computation method provided in the first aspect and / or the neural network training method provided in the third aspect.

[0064] Eighthly, embodiments of this application provide an electronic device including a processor configured to support the electronic device in implementing corresponding functions of the neural network computing method provided in the first aspect and / or the neural network computing method provided in the third aspect. The electronic device may further include a memory coupled to the processor, which stores necessary program instructions and data of the electronic device. The electronic device may also include a communication interface for communication between the neural network computing device and other devices or communication networks.

[0065] In a ninth aspect, embodiments of this application provide a computer storage medium for storing program code, the program code including methods for performing any of the methods described in the first or third aspect.

[0066] In a tenth aspect, embodiments of this application provide a computer program product including instructions that, when executed by a computer, enable the computer to perform any of the methods described in the first or third aspect.

[0067] Eleventhly, this application provides a chip system including a processor for supporting a neural network application device to implement the functions involved in the first aspect above, or for supporting a neural network training device to implement the functions involved in the third aspect above. For example, performing convolution calculations on the input data based on the weights of the target convolutional layer to obtain first output data, and calculating on the first output data based on the attention vector corresponding to the target convolutional layer to obtain second output data. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the data transmission device. This chip system can be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0069] Figure 1 This is a schematic diagram of a neural network provided in an embodiment of this application.

[0070] Figure 2 This is a schematic diagram of a memristor cross array provided in an embodiment of this application.

[0071] Figure 3 This is a schematic diagram of another memristor cross array provided in an embodiment of this application.

[0072] Figure 4 A schematic diagram of a convolutional neural network is provided for an embodiment of this application.

[0073] Figure 5 Another schematic diagram of a convolutional neural network is provided for the embodiments of this application.

[0074] Figure 6 This is a system architecture diagram provided in an embodiment of this application.

[0075] Figure 7 This is a hardware structure diagram of a neural network processor provided in an embodiment of this application.

[0076] Figure 8 This is a flowchart illustrating a neural network training method provided in an embodiment of this application.

[0077] Figure 9 This is a schematic diagram of a neural network training method provided in an embodiment of this application.

[0078] Figure 10 This is a flowchart illustrating another neural network training method provided in an embodiment of this application.

[0079] Figure 11 This is a schematic diagram of a neural network training method provided in an embodiment of this application.

[0080] Figure 12 This is a schematic diagram comparing the computational performance of the neural network in this embodiment with that of the neural network in the prior art.

[0081] Figure 13 This is a flowchart illustrating a neural network computation method provided in an embodiment of this application.

[0082] Figure 14 This is a schematic diagram illustrating the mathematical expression of a neural network calculation method provided in an embodiment of this application.

[0083] Figure 15 This is a flowchart illustrating a neural network training and calculation method provided in an embodiment of this application.

[0084] Figure 16 This is a schematic diagram of the structure of a chip system provided in an embodiment of this application.

[0085] Figure 17 This is a schematic diagram of the structure of a neural network training device provided in an embodiment of this application.

[0086] Figure 18 This is a schematic diagram of the structure of a neural network computing device provided in an embodiment of this application.

[0087] Figure 19 This is a schematic diagram of another neural network training device provided in an embodiment of this application.

[0088] Figure 20 This is a schematic diagram of another neural network computing device provided in an embodiment of this application. Detailed Implementation

[0089] The embodiments of this application will now be described with reference to the accompanying drawings.

[0090] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0091] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0092] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0093] The neural network computing method and apparatus provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0094] To facilitate understanding of the embodiments of this application, the following detailed analysis addresses the technical problems that need to be solved by the embodiments of this application and the corresponding application scenarios.

[0095] The building blocks of a neural network are neurons, and a large number of neurons are interconnected to form a network. The connections between neurons can be viewed as weighted directed edges. The output of each neuron is weighted by the connections between neurons and then passed to the neurons it is connected to. All the inputs received by each neuron are accumulated and further processed to produce the neuron's output. Neural network models are typically built by connecting layers of several neurons together, such as... Figure 1 The diagram shown is a schematic of a chain-like neural network. Figure 1 Each circle in the diagram represents a neuron, and each arrow represents a connection between neurons, with each connection having a weight. Of course, this embodiment can be applied to neural networks of any shape. Figure 1 The chain-like neural network shown is merely an illustration and is not intended to limit the application scenarios of the neural network training method provided in this application.

[0096] For example, a neural network has a layer L_n with n neurons and a layer L_m with m neurons, both fully connected (i.e., each neuron in layer L_n is connected to each neuron in layer L_m). The output of layer L_n is input to layer L_m after weighted connections. The input of layer L_n can be represented by a vector V_n of length n, and the connection weights can be represented by a matrix M_(n×m), with a size of n rows and m columns, where each matrix element represents the weight of a connection. Therefore, the weighted output of layer L_n, i.e., the vector input from layer L_n to layer L_m, is M_(n×m)V_n. This matrix-vector multiplication operation is the core and most common computation in neural networks.

[0097] A memristor, short for memory resistor, is a circuit device that represents the relationship between magnetic flux and electric charge. While a memristor has the dimension of resistance, unlike a regular resistor, its resistance value is determined by the charge flowing through it. Therefore, as a non-linear circuit element with resistive memory behavior, a memristor can store data using its resistance value; that is, by measuring the resistance of a memristor, the amount of charge flowing through it can be determined, thus achieving the function of remembering charge. Furthermore, the data stored by a memristor is non-volatile, meaning its resistance remains unchanged after power is turned off.

[0098] The matrix-vector multiplication and addition operations of large-scale neural networks pose significant challenges to applications with high real-time requirements and power sensitivity. How to process data quickly and efficiently using computer hardware has become a critical issue. Central Processing Units (CPUs) are limited by the bandwidth constraints of serial processing and data retrieval when performing matrix-vector multiplication, resulting in low computational efficiency. While Graphics Processing Units (GPUs) offer higher parallel computing efficiency for matrix-vector multiplication, they consume significant energy. Therefore, existing general-purpose processors are insufficient to meet the computational demands of deep learning, making the design of dedicated chips an important development direction. Since memristors possess integrated storage and computation capabilities, simulating matrix-vector multiplication using memristor cross arrays has become an effective method to solve these problems. A memristor cross array, or simply memristor array, consists of horizontally and vertically distributed metal lines and memristors at the intersections of these lines. The emergence of memristor cross arrays provides an efficient solution for neural network chip design, featuring high density, non-volatility, low power consumption, and integrated storage and computation. It can simultaneously store and compute neural network weights to efficiently perform multiplication and addition calculations within the neural network. Figure 2 A schematic diagram of a memristor cross array is shown below. Figure 2 As shown, when different voltage amplitudes are input to each row of the cross array, the memristors in the same column will convert the corresponding voltages into currents and sum them for output. Specifically, the input voltage and the conductance of the memristors are multiplied by a matrix to obtain the output current. This is equivalent to efficiently performing matrix-vector multiplication in an analog manner. In this case, the memristor cross array is responsible for both storing the matrix and calculating the matrix-vector multiplication.

[0099] Optionally, to convert the summed current into a voltage output, a load resistor is connected to the output port of each column. The external circuit only needs to read the voltage value across the load resistor. This feature of the memristor cross array has high application value and can greatly improve the computational efficiency of neural networks. The input voltage corresponds to the input data in the neural network, the memristor conductance represents the weights of the neural network, and the output current equals the input voltage multiplied by the conductance value, representing the output of the neural network. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of another memristor cross array provided in an embodiment of this application. The input vector consists of different voltage values ​​V0-V0. n This can be represented as a vector V. After calculation by the memristor cross-switch array, the output is a new voltage value V'0-V'. m This can be represented as a vector V', where V' = VGR. s Vector R s Including elements ,in This represents the grounding resistance value in column j. Figure 3The conductance G of the memristor switch array shown can be represented by the weight G_((n+1)×(m+1)) as shown below.

[0100]

[0101] Where, matrix G_((n+1)×(m+1)) is an (n+1) row (m+1) column matrix, and matrix element G ij This represents the conductance value in the i-th row and j-th column. For example, V'0 = (V 0* G 0,0 +V 1* G 1,0 +…+V n* G n,0 ).

[0102] The neural network training method provided in this application embodiment can be applied to, for example... Figure 2 or Figure 3 The memristor cross array shown.

[0103] However, due to manufacturing limitations of memristors, in practical applications, memristor devices suffer from low accuracy and large perturbations. Specifically, during actual operation, read and write errors cause the memristor's conductance value to not precisely match the required value, but rather fall within a distribution around the expected value, ultimately leading to decreased inference accuracy. In other words, each element G in the weights G_((n+1)×(m+1)) ij The value is not exactly equal to the required conductance value for each of the memristor switches in the array. The noise present in the memristor device negatively impacts the expressive power of memristor-based neural networks.

[0104] In existing technologies, to eliminate the impact of noise on the performance of actual neural networks, simulated noise from the circuit components in the neural network system can be incorporated during neural network training. By exposing the noise of the underlying components to the software training layer, some of the accuracy impact caused by hardware-level noise can be offset. For example, during neural network training, when performing input-weight multiplication, noise is added to each element of the weights. Optionally, this noise can be a normally distributed noise to simulate memristor device noise. This allows the neural network training to adapt to the presence of noise, rather than just providing an accurate weight representation. While the above method introduces noise into the network training process, increasing the robustness of the network weights to some extent, it only resists noise to a certain degree. Its effect is minimal against the significant noise generated by actual memristor devices, and it does not reduce power consumption. Moreover, when the actual memristor device noise is large, the noise introduced into the neural network during training cannot converge. Therefore, how to improve the accuracy of neural networks and reduce the impact of noise from circuit components (such as memristors) in the neural network system on the accuracy of the neural network is an urgent problem to be solved.

[0105] This application provides a neural network training and computation method. An attention vector is introduced during the neural network training and practical application process. Specifically, in a target convolutional layer of the neural network, the input data of the target convolutional layer is further calculated using the weights of the target convolutional layer to obtain first output data. Based on the attention vector corresponding to the target convolutional layer, the first output data is then calculated to obtain second output data, which becomes the input data for the next layer of the target convolutional layer. In obtaining the input data for the next layer of the target convolutional layer, in addition to extracting feature maps from the input data using weights, the extracted feature maps are also modified using the attention vector. For example, feature maps with strong robustness to noise are strengthened, while feature maps with poor robustness to noise are weakened or even removed, thereby reducing the impact of noise on the neural network system and improving the accuracy of neural network computation. This neural network training method can be executed by a neural network training device, and the neural network computation method can be executed by a neural network computation device. The neural network training device or neural network computation device can be a chip or a chip system; it can also be a computer-readable storage medium; it can also be a computer program product; this application does not limit this.

[0106] It should be noted that Figure 1 The neural network architecture shown and Figure 2 and Figure 3The illustrated diagram of the memristor cross array is for illustrative purposes only and is not intended to limit the technical solutions of this application. Those skilled in the art should understand that in specific implementations, the neural network architecture can take other forms, may include other devices, and the number of memristors can be configured according to specific needs.

[0107] The following explanations of some of the terms used in neural networks will be provided to facilitate understanding by those skilled in the art.

[0108] (1) Deep Neural Networks (DNN) is a broad concept that, in a sense, includes Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Generative Adversarial Networks (GAN). A DNN refers to a neural network containing multiple hidden layers. The neural network computation method and related devices provided in this application can be applied to deep neural networks, especially to convolutional neural networks.

[0109] (2) Convolutional Neural Network (CNN) is a multi-layered neural network. Each layer consists of multiple two-dimensional planes, and each plane consists of multiple independent neurons. The multiple neurons in each plane share weights. By sharing weights, the number of parameters in the neural network can be reduced. Currently, in CNNs, the processor usually performs convolution operations by convolving the input signal features with the weights, which is converted into matrix multiplication between the signal matrix and the weights.

[0110] (3) Convolutional kernels can be initialized as matrices of random size. During the training of a convolutional neural network, the kernels can learn to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the connections between layers of the convolutional neural network, while also reducing the risk of overfitting.

[0111] (4) A filter is a concatenation of multiple convolutional kernels, each assigned to a specific channel of the input. When the number of channels is 1, the filter is the convolutional kernel; when the number of channels is greater than 1, the filter refers to a concatenation of multiple convolutional kernels. For example, if an image is stored as a tensor in RGB format, the input includes three channels: the R matrix, the G matrix, and the B matrix (red, green, and blue, corresponding to three images of the same size). Each channel's matrix is ​​convolved with its corresponding convolutional kernel, and all the convolutional kernels corresponding to all channels form the filter. Each filter is used to extract different feature data. For example, an image has four channels (ARGB) (transparency and red, green, and blue, corresponding to four images of the same size). Assuming the convolution kernel size is 100*100, a total of 16 convolution kernels w1 to w16 are used. Kernel w1 to w4 form the first filter, kernels w5 to w8 form the second filter, kernels w9 to w12 form the third filter, and kernels w13 to w6 form the fourth filter. Different filters are used to extract different feature data from the input image. Performing a convolution operation on the ARGB image using the first filter, that is, using w1 to w4 corresponding to the four images on the four channels, yields the first image. The top-left pixel of this image is the weighted sum of the pixels within a 100*100 area in the top-left corner of the four input images, and so on. Similarly, including the other filters, the output of this layer corresponds to four "images." Each image pair is a response to different features in the original image.

[0112] (5) Convolutional neural networks can use the back propagation (BP) algorithm to correct the parameters in the initial neural network model during training, thereby reducing the reconstruction error loss of the neural network model. Specifically, forward propagation of the input signal to the output will generate error loss. By backpropagating the error loss information, the parameters in the initial neural network model are updated, thus converging the error loss. The back propagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weights and attention vectors in the embodiments of this application.

[0113] (6) Convolution is the extraction of feature data from the original input. Simply put, feature extraction involves extracting features from small regions of the original input. Mathematically, convolution is the operation between the convolution kernel and the input matrix of the convolutional layer. The input matrix is ​​usually extracted from the image matrix during convolution based on the stride of the convolution kernel. The convolution kernel is a small window that records the weights. The convolution kernel slides across the image matrix with the stride. Each slide corresponds to a submatrix of the image matrix, and the weights in the kernel are multiplied and summed with the values ​​in the submatrix, then assigned to an element in the current output feature map (output matrix). Convolution is not limited to the original input; it also includes re-convolution on the output result after convolution. This application does not impose such limitations. For example, the first convolution extracts low-level feature data, the second convolution extracts mid-level feature data, the third convolution extracts high-level features, and so on. Features can be continuously extracted and compressed. The higher-level features obtained at the end can be understood as a further condensation of the original features, making the final features more reliable. The last layer of features can be used to process various tasks, such as classification and regression.

[0114] (7) Convolution is one of the most important operators in convolutional neural networks. For example, X represents the input feature map (the input matrix of the convolutional layer), W represents the weights, b represents the bias, Y0 represents the product of the X and W matrices, and Y represents the output feature map (the output matrix of the convolutional layer). Optionally, after an optional activation operation, the activation value of each element in the output Y is calculated to obtain the final result.

[0115] (8) Robustness refers to the characteristic that a system, device, or apparatus can maintain certain other performance characteristics under certain parameter perturbations (structure, size). The main causes of parameter perturbations include the slow drift of characteristics or parameters caused by environmental factors (such as noise) during the operation of the system, device, or apparatus.

[0116] like Figure 4 As shown, Figure 4 This is a schematic diagram of a convolutional neural network provided in an embodiment of this application. The convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, fully connected layers 131 to 13n and an output layer 140, wherein the pooling layer is optional.

[0117] Convolutional / pooling layers 120:

[0118] like Figure 4The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0119] Convolutional layers:

[0120] Taking convolutional layer 121 as an example, convolutional layer 121 can include many convolution operators, also called convolution kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input matrix. Essentially, a convolution operator can be a weight, which is usually predefined. During the convolution operation on the image, the weight is typically processed pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction of the input matrix, thereby extracting specific feature data from the input matrix. When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general feature data, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, later convolutional layers (e.g., 126) extract increasingly complex features, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem being solved.

[0121] Pooling layer:

[0122] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 1 In the example of 120, layers 121-126 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image.

[0123] Fully connected layers 131 to 13n and output layer 140:

[0124] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract feature data and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize fully connected layers 131 to 13n and the output layer 140 to generate one or more of the required class outputs.

[0125] It should be noted that, as Figure 4 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of another convolutional neural network provided in an embodiment of this application. Multiple convolutional / pooling layers run in parallel, and the extracted features are all input to fully connected layers 131 to 13n for processing.

[0126] It should be noted that Figure 4 or Figure 5 The neural network architecture shown is for illustrative purposes only and is not intended to limit the technical solutions of this application. Those skilled in the art should understand that in specific implementations, the neural network architecture can take other forms, may include other devices, and the number of memristors can be configured according to specific needs.

[0127] See appendix Figure 6 , Figure 6 This application provides a system architecture. As shown in the system architecture, the system may include a neural network computing device 110, a neural network training device 120, a database 130, a client device 140, a data storage system 150, and a data acquisition device 160, etc. The data acquisition device 160 is used to collect training data. In this application, the data acquisition device 160 may include, for example, a microphone or a camera. The training data (i.e., the input and output of the neural network training side) in this application embodiment may include, for example, video sample data, image sample data, or voice sample data, and targets (or user tags) matching the aforementioned video sample data, image sample data, or voice sample data. For example, the image sample data is a user's facial image, and the target corresponding to the image sample data is the user's identity information. The aforementioned video sample data, image sample data, or voice sample data, and the targets or tags matching the aforementioned video sample data, image sample data, or voice sample data, may be collected by the data acquisition device 160 or downloaded from the cloud. Figure 6This is merely an exemplary architecture and is not intended to limit the scope of the invention. Optionally, the training data in this embodiment may also include other types of sample data besides video sample data, sample data, or audio sample data; this embodiment does not impose any limitations. Further, the data acquisition device 160 stores the training data in the database 130, and the neural network training device 120 trains the network parameters of the neural network system based on the training data maintained in the database 130 (the network parameters of the neural network system here may be network parameters trained using the neural network training method described in this embodiment, such as weights and attention vectors).

[0128] It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, the neural network training device 120 may not necessarily be trained entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0129] The neural network computing device 110 or neural network training device 120 in this application embodiment can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, smart wearable device, smart robot, vehicle terminal, etc., or it can be a server or cloud, etc. (See attached...) Figure 6 In this embodiment, the neural network computing device 110 is equipped with an I / O interface for data interaction with external devices. Users can input data into the I / O interface through a client device 140 (the client device in this application may also include data acquisition devices such as microphones and cameras). The input data (i.e., the input data on the computing side) may include voice information, image information, or video information. It is understood that the input data here may be user-input data or data provided by a relevant database, depending on the application scenario. This embodiment does not specifically limit this.

[0130] In this embodiment, the client device 140 and the neural network computing device 110 can be on the same device, and the data acquisition device 160, the database 130 and the neural network training device 120 can also be on the same device as the neural network computing device 110 and the client device 140.

[0131] It is worth noting that the neural network training device 120 can generate network parameters of the corresponding neural network system based on different training data for different goals or tasks. The corresponding neural network system can then be used to achieve the aforementioned goals or complete the aforementioned tasks, thereby providing the user with the desired results.

[0132] It is worth noting that, attached Figure 6 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 6 In this embodiment, the data storage system 150 is an external memory relative to the neural network computing device 110. However, in other cases, the data storage system 150 may be placed within the neural network computing device 110. (Appendix) Figure 6 The specific structure of the neural network training device 120 can be found in the appendix of the embodiments of this application. Figure 17 Or attached Figure 19 , attached Figure 6 The specific structure of the neural network computing device 110 can be found in the appendix of the embodiments of this application. Figure 18 Or attached Figure 20 However, it is worth noting that... Figure 17 Or attached Figure 19 The disclosed structure is merely an illustrative structure of the neural network training device 120 proposed in the embodiments of this application. Figure 18 Or attached Figure 20 The disclosed structure is merely an illustrative structure of the neural network computing device 110 proposed in the embodiments of this application, and the embodiments of this application do not limit it.

[0133] Based on the above Figure 4 and Figure 5 Description of the relevant functions of convolutional neural networks and Figure 6 For a description of the system architecture, please refer to [link / reference]. Figure 7 , Figure 7 This is a hardware structure diagram of a neural network processor provided in an embodiment of this application, wherein:

[0134] The neural network processor (NPU) 302 is mounted as a coprocessor on the CPU (such as the host CPU) 301, and tasks are assigned by the host CPU 301. For example, in the system architecture described above, the CPU 301 can be located in the client device 140, used to extract the voice information or image information to be recognized from the voice data and video data; while the NPU 302 can be located in the computing module 111, used to perform feature extraction and feature matching on the voice information or image information to be recognized extracted by the CPU 301, and then send the matching result to the CPU 301 for further computational processing, which will not be described in detail here. It is understood that the CPU and NPU can be located in different devices, and their configurations can be different depending on the actual needs of the product. For example, the NPU is located on a cloud server, while the CPU is located on a user device (such as a smartphone or intelligent robot); or, both the CPU and NPU are located on the client device (such as a smartphone or intelligent robot).

[0135] The core of the NPU 302 is the arithmetic circuit 3023, which is controlled by the controller 3024 to extract matrix data from the memory and perform multiplication operations.

[0136] In some implementations, the arithmetic circuit 3023 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 3023 is a two-dimensional pulsating array. The arithmetic circuit 3023 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 3023 is a general-purpose matrix processor. For example, suppose there is an input matrix A and weights B. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 3022 and caches the corresponding data of matrix B in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 3021 and performs matrix operations with matrix B. The partial result or the final result of the matrix is ​​stored in the accumulator 3028.

[0137] Unified memory 3026 is used to store input and output data. Weight data is directly accessed via the Direct Memory Access Controller (DMAC) 3025 and then transferred to the weight memory 3022. Input data is also transferred to unified memory 3026 via DMAC.

[0138] The Bus Interface Unit (BIU) 30210 is used by the instruction fetch memory 3029 to fetch instructions from external memory, and also by the memory access controller 3025 to fetch the original data of the input matrix A or weights B and attention vector V from external memory.

[0139] The DMAC is mainly used to move input data from external memory DDR to unified memory 3026, or to weight data to weight memory 3022, or to input data to input memory 3021.

[0140] The vector calculation unit 3027 may include multiple operation and processing units. The vector calculation unit 3027 is used to retrieve the attention vector stored in the vector memory 3030 and perform recalculation based on the matrix operation result obtained by the operation circuit 3023 and the attention vector.

[0141] The instruction fetch buffer 3029 connected to the controller 3024 is used to store the instructions used by the controller 3024;

[0142] Unified memory 3026, input memory 3021, weight memory 3022, vector memory 3030, and instruction fetch memory 3029 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0143] It is understood that the functions related to the extraction and correction of feature data in the input in any embodiment of the neural network training method in this application, as well as the extraction and correction of feature data in the input in any embodiment of the neural network calculation method in this application, are all implemented by the relevant functional units in the aforementioned neural network processor 302 (NPU), and will not be described in detail here.

[0144] The following describes embodiments of the neural network training method and neural network computation method provided in this application from the perspectives of model training and computation, based on the above application scenarios, system architecture, structure of convolutional neural networks, and structure of neural network processors. It also provides a detailed analysis and solution to the technical problems raised in this application.

[0145] Please see Figure 8 , Figure 8 This is a schematic flowchart of a neural network training method provided in an embodiment of this application, which can be applied to the above. Figure 7 The corresponding neural network processor. The method may include the following steps S801 to S804, and optionally, may also include steps S805 to S809.

[0146] Step S801: Input the training data into the neural network system, wherein the neural network system includes at least one convolutional layer.

[0147] Specifically, neural network systems update network parameters through gradient-based backpropagation. The required dataset typically consists of training and testing data. Generally, the neural network is given input data X (i.e., the training data in this embodiment), and then a target output data Y_label is defined, also known as the label. Based on the input data X, a set of output data Y is obtained. This Y is compared with the Y_label corresponding to X, and the difference between them (also called the loss value) is used to adjust the network parameters (gradient backpropagation parameter adjustment) so that the next output data Y is closer to the Y_label. After training, the trained network is further validated using test data, i.e., using data that the neural network has never used before to test the network and evaluate its accuracy on the test data. Higher test accuracy indicates better training of the neural network.

[0148] Step S802: The target convolutional layer calculates the input data of the target convolutional layer using the first attention vector and L first weights to obtain the input data of the next layer of the target convolutional layer.

[0149] The at least one convolutional layer includes the target convolutional layer. The L first weights are used for convolution calculation to extract feature data (or feature maps) from the input data of the target convolutional layer, and the first attention vector is used to correct the feature data, where L is an integer greater than or equal to 1.

[0150] Specifically, a convolutional layer can correspond to one or more filters, and a filter can be used for convolution calculation to extract a feature map from the input data of the convolutional layer. The input data of the convolutional layer can correspond to multiple channels. Taking an RGB image as an example, the input data of the convolutional layer corresponds to 3 channels. A filter corresponds to multiple convolutional kernels (i.e., weights), and each of the multiple convolutional kernels corresponds to each of the multiple channels. Therefore, when a filter extracts a feature map, it is actually by extracting features at the same position through the convolutional kernels corresponding to each channel and then superimposing them. In this embodiment, taking an RGB image as an example again, if there is only one filter, the target convolutional layer extracts a feature data from the input data of the target convolutional layer through 3 first weights. Further, after using L first weights for convolution calculation to extract the feature data from the input data of the target convolutional layer, a first attention vector is used to correct the feature data. The meaning of "correction" here can be either to strengthen the features that are robust to noise and weaken the features that are not robust to noise by using different coefficients; or to select the features that are robust to noise and remove the features that are not robust to noise; or to select the features that are robust to noise and assign different weighting coefficients to different features based on their different robustness to noise, while removing the features that are not robust to noise.

[0151] Step S803: Input the training data into the neural network to obtain the third output data and compare it with the target output data to calculate the first loss value.

[0152] Specifically, the process of calculating the third output data based on the training data and the network parameters of the neural network system can be called "forward propagation". Commonly used loss functions for calculating the loss value include, but are not limited to, the following: 0-1 loss function, quadratic loss function, absolute loss function, logarithmic loss function, and hinge loss function.

[0153] Step S804: Update the network parameters of the neural network system according to the first loss value, wherein the network parameters include the first attention vector and the L first weights.

[0154] The network parameters of the neural network system are the parameters that need to be trained in the neural network system, and may include at least the attention vector and weights corresponding to the convolutional layers. In this embodiment, the target convolutional layer corresponds to a first attention vector and the L first weights. However, this does not mean that all convolutional layers in the neural network system correspond to an attention vector to correct the feature data extracted by the weights. The neural network provided in this embodiment includes at least one convolutional layer. In this target convolutional layer, feature data is extracted from the input data of the convolutional layer based on multiple weights, and then the feature data is corrected by an attention vector. Specifically, in the process of updating the network parameters of the neural network system according to the first loss value, the first loss value is used as input data for backpropagation, which updates the network parameters of the neural network system, thus completing one training process of the neural network model. Then, the updated network parameters can be used as the network parameters for the next forward propagation, and steps S801 to S804 are continued until the first loss value is less than a preset threshold or the number of neural network training cycles reaches a preset number of iterations. Optionally, if the first loss value does not converge, backpropagation is performed based on the first loss value to update the first attention vector and L first weights. The first loss value is then re-determined based on the training data, the target output data, the updated first attention vector, and the updated L first weights, until the first loss value converges. When the first loss value converges, the current first attention vector and L first weights are determined to be the second attention vector and L second weights corresponding to the target convolutional layer. After multiple executions of the neural network model training process, the desired network parameters can be obtained, and the final network parameters can then be deployed into the circuit device.

[0155] In this embodiment, within a target convolutional layer of the neural network, the input data of the target convolutional layer is calculated using a first attention vector and L first weights to obtain the output data of the target convolutional layer. This output data serves as the input data for the next layer of the target convolutional layer. In training the neural network, this embodiment trains not only the weights but also the first attention vector to evaluate the noise robustness of each feature data extracted through weight convolution. Specifically, in obtaining the input data for the next layer of the target convolutional layer, in addition to extracting feature data from the input data using L first weights, the extracted feature data is also modified using the first attention vector. This strengthens feature data with strong noise robustness to the circuit components in the neural network system, weakens or even removes feature data with poor noise robustness to the circuit components, and selects the optimal feature data for the calculation of the next layer of the target convolutional layer. This prevents feature data with poor noise robustness to the circuit components in the neural network system from flowing into the next layer of the target convolutional layer, thereby reducing the impact of noise from the circuit components in the neural network system on the neural network system and improving the accuracy of the neural network.

[0156] In one possible implementation, before the target convolutional layer calculates the input data of the target convolutional layer using a first attention vector and L first weights, steps S805 and S806 may be included.

[0157] Step S805: Obtain the initial attention vector corresponding to the target convolutional layer.

[0158] Step S806: For the initial attention vector, set the elements in the vector that are less than or equal to the preset threshold to zero, and keep the elements in the vector that are greater than the preset threshold unchanged, to obtain the first attention vector.

[0159] Specifically, the weights and attention vector can be initialized before training the network parameters of the neural network system. For the initial attention vector, a portion of its elements are set to invalid values, i.e., a pruning operation is performed on the initial attention vector. In this embodiment, the elements to be pruned are set to zero, that is, elements in the vector less than or equal to a preset threshold are set to zero, while elements in the vector greater than the preset threshold remain unchanged, thus obtaining the first attention vector. The preset threshold can be determined by: sorting the elements in the initial attention vector in descending order of value; determining the value of the Nth element after sorting as the preset threshold, where N is based on a preset pruning rate, which represents the ratio of the number of invalid elements in the vector to the total number of elements in the vector. For example, a pruning rate of 20% means that 20% of the elements in the initial attention vector are pruned; specifically, the values ​​of 20% of the elements in the initial attention vector are set to zero. In this embodiment, the feature data in the input data of the target convolutional layer extracted by L first weights may include M feature maps, where M is an integer greater than or equal to 1 and less than or equal to L. The M feature maps correspond to M weight sets, and the union of the M weight sets is the L second weights. One weight set is used for convolution calculation to extract a feature map from the input data of the target convolutional layer. The initial attention vector includes M elements, and there is a one-to-one correspondence between the M weight sets, the M feature maps, and the M elements. By setting some elements in the initial attention vector to zero, the feature maps corresponding to elements with zero values ​​are removed, thereby removing feature data with poor noise robustness and retaining feature data with good noise robustness to enter the next layer of the target convolutional layer for calculation. This allows for the selection of the optimal channel combination (where each channel corresponds to a feature map) for inference and computation in the neural network system.

[0160] In one possible implementation, after obtaining the initial attention vector corresponding to the target convolutional layer, step S807 may also be included.

[0161] Step S807: Process the initial attention vector based on the activation function, and then set the elements in the vector that are less than or equal to the preset threshold to zero, while keeping the elements in the vector that are greater than the preset threshold unchanged, to obtain the first attention vector.

[0162] Commonly used activation functions include, but are not limited to, the Sigmoid function, tanh function, ReLU function, LeakyReLU function (PReLU), ELU (Exponential Linear Units) function, and MaxOut function. The Sigmoid function is a commonly used non-linear activation function, and its mathematical form is: The Sigmoid function transforms continuous real values ​​of input data into output data between zero and 1. Specifically, if the input is a very large negative number, the output data is 0; if the input is a very large positive number, the output data is 1. The non-zero elements of the first attention vector after activation function processing are located in a closed interval between 0 and 1, which can improve the performance of the neural network and make it easier for the neural network to converge during training. The preset threshold in this embodiment can be determined as follows: The elements in the initial attention vector after activation function processing are sorted in descending order of value; the value of the Nth element after sorting is determined as the preset threshold, where N is based on a preset pruning rate, which represents the ratio of the number of invalid elements in the vector to the total number of elements in the vector. Elements in the vector less than or equal to the preset threshold are set to zero, while elements greater than the preset threshold remain unchanged, thus obtaining the first attention vector.

[0163] In one possible implementation, steps S808 and S809 may be included before the target convolutional layer is used to compute the input data of the target convolutional layer using a first attention vector and L first weights.

[0164] Step S808: Obtain the L initial weights of the target convolutional layer.

[0165] As mentioned above, the weights and attention vectors can be initialized before training the network parameters of a neural network system. The initialization of weights in a neural network model is crucial for training; poor initialization can lead to gradient propagation problems and slow down training, while good initialization can accelerate convergence and increase the likelihood of finding a better solution. If the weights are initially small, the signal will also be small at the end; if the weights are initially large, the signal will also be large at the end. Common weight initialization methods include, but are not limited to, the following: 1. Constant initialization: Initializing the elements of the weights to a user-defined constant; 2. Gaussian initialization: Initializing the elements of the weights to small random numbers, such as a Gaussian distribution with a mean of zero and a variance of zero.01. This initialization method is only suitable for small networks. For deep networks, small weights result in small gradients during backpropagation, thus weakening the gradient "signal"; 3. Positive unitball initialization; 4. Initializing the elements of the weights to follow a normal distribution; 5. Uniform initialization: Initializing the elements of the weights to follow a uniform distribution, using a maximum and a minimum value to control the upper limit of the element values; 6. Xavier initialization: The elements of the weights follow a uniform distribution with a mean of zero and a variance correlated with the input data; 7. MSRA initialization: The elements of the weights follow a Gaussian distribution with a mean of zero and a variance correlated with the input data, etc.

[0166] Step S809: Introduce the read / write error of the circuit device in the simulated neural network system into the L initial weights to obtain the L first weights.

[0167] The L initial weights correspond one-to-one with the L first weights. Optionally, the read / write errors of the circuit devices in the simulated neural network system follow a normal distribution, and the standard deviation of the normal distribution is related to the magnitude of the read / write errors of the circuit devices in the neural network system.

[0168] Specifically, a noise is added to each element of the L initial weights. This noise simulates the noise of circuit devices in a neural network system, such as memristors. Optionally, the noise of the circuit devices in the neural network system can follow a normal distribution, and the standard deviation of the normal distribution is related to the magnitude of the read / write error of the circuit devices in the neural network system. In other embodiments, the noise of the circuit devices in the neural network system can also follow other distributions, and the distribution of the noise of the circuit devices in the neural network system is related to the circuit devices in the neural network system. This application embodiment does not impose any limitations. In this application embodiment, during the training of the L first weights, the simulated noise of the circuit devices in the neural network system can be combined during neural network training. By exposing the noise of the underlying devices to the software training layer, the accuracy impact caused by some of the hardware-level noise can be offset, thereby enabling the training of the neural network to adapt to the presence of noise, rather than just a precise weight representation. Therefore, this application embodiment introduces noise into the network training process, which to a certain extent increases the robustness of the weights.

[0169] In one possible implementation, updating the network parameters of the neural network system based on the first loss value includes: updating the initial attention vector and L initial weights based on the first loss value; updating the L first weights based on the updated L initial weights; and updating the first attention vector based on the updated initial attention vector.

[0170] In this embodiment, when updating the L first weights and the first attention vector, the initial attention vector and the L initial weights are updated via backpropagation based on the first loss value. For the updated initial attention vector (or the initial attention vector processed by the activation function), elements in the vector less than or equal to the preset threshold are set to zero, while elements in the vector greater than the preset threshold remain unchanged, thus updating the first attention vector. For the updated L initial weights, when performing the input data multiplication operation with the weights, noise is added to each element of the initial weights. Optionally, this noise can be a normally distributed noise, used to simulate the noise of circuit devices in neural network systems such as memristors, thereby obtaining the updated L first weights.

[0171] The trained neural network model was tested using test data. During the testing process, it was observed that the first attention vector filters the results of the convolutional layer calculations. Specifically, Figure 9 This is a schematic diagram of a neural network training method provided in an embodiment of this application, as shown below. Figure 9As shown, after the input data of the target convolutional layer is convolved with L first weights corresponding to the target convolutional layer, features 1 to 6 are extracted. The attention vector after processing by the activation function corresponding to the target convolutional layer can be (0.2, 0.3, 0.4, 0.9, 0.6, 0.8). Since the value of each element in the vector represents the robustness of the corresponding feature data to noise, it can be seen that features 1 to 3 have poor robustness to noise and do not benefit the lower layer calculation of the target convolutional layer, so the values ​​of the elements in their corresponding attention vectors are set to zero; features 4 to 6 have good robustness to noise and can be retained for the lower layer calculation of the target convolutional layer. Furthermore, for the retained feature data, it is multiplied by the value of the corresponding element in the attention vector, and the corresponding features are corrected by multiplying by different element values ​​corresponding to different elements.

[0172] The neural network corresponding to the network parameters of the neural network system trained based on the above steps is labeled as neural network 1. The neural network 1 trained by this invention is experimentally verified on voice wake-up data. Table 1 compares the wake-up rate of neural network 1 trained by the neural network training method described in the embodiments of this application with that of neural networks in the prior art under the same false wake-up rate. "Std3" and "Std6" are used to represent the noise present in the circuit devices of current memristor and other neural network systems. "Std3" represents the noise following a normal distribution of 0 to 3 added to the neural network during the test, and "Std6" represents the noise following a normal distribution of 0 to 6 added to the neural network during the test. The noise simulated by "Std6" is greater than the noise simulated by "Std3".

[0173] Table 1

[0174]

[0175] As can be seen from the table above, the neural network trained by the neural network training method described in the embodiments of this application has an improvement of about one point compared with the neural network in the prior art when the standard value is 3. This is because the noise model is relatively small, and the attention vector may not be able to clearly distinguish the robust feature data. However, when the noise is increased, the neural network trained by the neural network training method described in the embodiments of this application has an improvement of about four points compared with the neural network in the prior art when the standard value is 6. In other words, the better the beneficial effect of the present invention is when the noise is relatively large.

[0176] Steps S801 to S809 described above can be summarized as training the network parameters of the neural network model through software. After training the second attention vector and L second weights through the software in the above steps, in one possible implementation, the second attention vector can be deployed to digital circuit devices (e.g., digital storage modules) in the neural network system, and the L second weights can be deployed to analog circuit devices (e.g., memristors) in the neural network system. For example, the weights of the L second weights can be set to the conductance values ​​corresponding to the cross array of the memristors. After training the second attention vector and L second weights through the software in steps S801 to S809, this embodiment of the application may also include steps S810 to S813. Steps S810 to S813 describe deploying the second attention vector and L second weights trained through the software in steps S801 to S809 to the corresponding hardware, inputting the training data back into the neural network, and retraining the second attention vector in the network parameters of the neural network system by combining the actual noise of the circuit devices in the neural network system.

[0177] like Figure 10 As shown, Figure 10 This is a flowchart illustrating another neural network training method provided in an embodiment of this application. The neural network training method further includes steps S810 to S813. Wherein, the updated L first weights are replaced by L second weights, and the updated first attention vector is replaced by a second attention vector.

[0178] Step S810: Input the training data into the neural network system.

[0179] Step S811: The target convolutional layer calculates the input data of the target convolutional layer using the second attention vector and L third weights to obtain the input data of the next layer of the target convolutional layer.

[0180] Specifically, each of the L third weights corresponds one-to-one with one of the L second weights. Each third weight is obtained by writing the corresponding second weight into the circuit device of the neural network system and then combining it with the actual noise of the circuit device in the neural network system. After the network parameters trained by the software are deployed to the actual circuit device, the output data of the neural network system is obtained. The L second weights corresponding to the target convolutional layer written into the analog circuit device are converted into L third weights due to the influence of the noise of the circuit device in the neural network system during the actual circuit device process.

[0181] Step S812: Compare the fourth output data with the target output data and calculate the second loss value.

[0182] Specifically, the neural network system obtains the fourth output data by performing forward propagation based on the network parameters corresponding to the neural network deployed on the actual hardware. The second loss value can be calculated using loss functions such as, but is not limited to, 0-1 loss function, quadratic loss function, absolute loss function, logarithmic loss function, and hinge loss function.

[0183] Step S813: Update the second attention vector in the network parameters according to the second loss value.

[0184] Since the L second weights deployed to the circuit devices in the neural network system need to be erased before rewriting, for example, deploying the L second weights as the conductance values ​​corresponding to the memristor cross array will not change before erasure. Therefore, updating the neural network parameters again based on steps S810 to S813 is equivalent to fixing the weights in the network parameters of the neural network system, inputting the second loss value as input data into the neural network for backpropagation, and retraining the second attention vector. In the process of updating the network parameters of the neural network system according to the second loss value, the second loss value is used as input data for backpropagation, which updates the network parameters of the neural network system, thus completing one training process of the neural network model. Then, the updated attention vector can be used as the network parameters for the next forward propagation, and steps S810 to S813 can be continued until the second loss value is less than a preset threshold or the number of neural network training cycles reaches a preset number of iterations. Optionally, specifically, if the second loss value does not converge, backpropagation is performed based on the second loss value to update the second attention vector, and the second loss value is re-determined based on the training data, the target output data, and the updated second attention vector until the second loss value converges; it is determined that when the second loss value converges, the current second attention vector is the final attention vector corresponding to the target convolutional layer.

[0185] like Figure 11 As shown, Figure 11This is a schematic diagram of a neural network training method provided in an embodiment of this application. The neural network training method proposed in this embodiment first trains the network parameters of the neural network system in the software part, and then deploys the neural network model trained by the software onto the corresponding circuit devices and digital circuit devices in the neural network system through on-chip training. The weights already deployed on the circuit devices in the neural network system are fixed, and the second attention vector is optimized to obtain a new attention vector. Since it further fits the actual noise of the circuit devices in the neural network system, it can more accurately combine the feature data to correct the feature data based on the robustness of the feature data to noise.

[0186] The neural network corresponding to the network parameters of the neural network system trained based on steps S810 to S813 above is labeled as neural network 2. The neural network 2 trained in this application embodiment is still tested and verified on voice wake-up data. Table 2 compares the wake-up rate of the neural network trained by the neural network training method described in this application embodiment with that of the neural network in the prior art under the same false wake-up rate. Among them, "Std3" and "Std6" are used to represent the noise present in the circuit devices of the current memristor and other neural network systems. "Std3" is used to represent the noise that follows a normal distribution of 0 to 3 added to the neural network during the test, and "Std6" is used to represent the noise that follows a normal distribution of 0 to 6 added to the neural network during the test. The noise simulated by "Std6" is greater than the noise simulated by "Std3". The network parameters of neural network 1 are trained through the software training process represented by steps S801 to S809. The network parameters of neural network 2 are obtained by deploying the network parameters to the corresponding digital and analog circuit devices in the neural network system after the network parameters are trained by software, and then fitting them to the actual noise of the circuit devices in the neural network system. That is, they are obtained through the above steps S801 to S813.

[0187] Table 2

[0188]

[0189] Please refer to Figure 12 , Figure 12 This is a schematic diagram comparing the computational performance of the neural network in this application embodiment with that of the neural network in the prior art, in conjunction with Tables 1 and 2. Figure 12 It is known that after retraining the attention vector on the device, the performance of the embodiments of this application is improved by 1 point compared with the neural network 1 of the present invention that has not been retrained, and by about 5 points compared with the original network in the prior art, which is basically close to the performance of the original network in the art at std3.

[0190] The above embodiments only describe the training of the second attention vector and L second weights after training them in the software stage, followed by hardware training. This involves deploying the second attention vector and L second weights to the corresponding hardware, inputting the training data back into the neural network, and retraining the second attention vector in the network parameters of the neural network system by incorporating the actual noise of the circuit devices. It is understood that the above training process is merely an example and is not intended to limit the technical solutions of this application. For example, training only in the software stage (such as some or all of steps S801 to S809), or training only in the hardware stage (such as some or all of steps S810 to S813), or training in the hardware stage first and then in the software stage, or training in the software stage and hardware stage simultaneously, all fall within the scope of protection of this invention.

[0191] After training the neural network model through the above embodiments, inference calculations can be performed based on the trained neural network model.

[0192] Based on Figures 8 to 11 The neural network training method described in any of the corresponding method embodiments obtains the network parameters of the neural network system, and performs actual calculations of the neural network based on the obtained network parameters. See also Figure 13 , Figure 13 This is a flowchart illustrating a neural network computation method provided in an embodiment of this application. The neural network computation method describes the above-described... Figures 8 to 11 The actual application of the neural network system trained in any of the corresponding method embodiments. The neural network computation method is performed by a neural network system including at least one convolutional layer, and the method may include method flow steps S1301 to S1303.

[0193] Step S1301: Obtain the input data for the target convolutional layer.

[0194] Specifically, the neural network system includes at least one convolutional layer, and the target convolutional layer is any one of the at least one convolutional layers. That is, in all the convolutional layers included in the neural network system, each convolutional layer may correspond to a corresponding attention vector; alternatively, not every convolutional layer may correspond to a corresponding attention vector, but only at least one convolutional layer exists, and each of the at least one convolutional layer corresponds to an attention vector. The input data of the neural network system may include video data, image data, or audio data, etc., and this application embodiment does not impose any limitations.

[0195] Step S1302: Perform convolution calculation on the input data based on the weights of the target convolutional layer to obtain first output data, wherein the first output data includes N first feature maps, and N is an integer greater than or equal to 1.

[0196] Specifically, the weights of the aforementioned target convolutional layer can be based on... Figures 8 to 11 The neural network training method described in any of the corresponding method embodiments obtains the updated L first weights.

[0197] The number of feature maps corresponds to the number of channels in a convolutional layer and the number of filters between convolutional layers. Taking image data as the input data of a neural network system as an example, in the input layer, if the input data is a grayscale image, there is only one feature map; if the input data is an RGB image, there are generally three feature maps. In the convolutional layers, there are several filters between convolutional layer 1 and convolutional layer 2. A filter is a concatenation of multiple convolutional kernels, each kernel assigned to a specific channel of the input. When the number of channels is one, the filter is the convolutional kernel; when the number of channels is greater than one, the filter refers to a concatenation of multiple convolutional kernels. For example, if an image is stored as an RGB tensor, the input includes three channels: the R matrix, the G matrix, and the B matrix (red, green, and blue, corresponding to three images of the same size). Each channel's matrix is ​​convolved with a corresponding convolutional kernel, and all the convolutional kernels corresponding to all channels form a filter. Each filter is used to extract different feature maps (or feature data). For example, an image has four channels (ARGB) (transparency and red, green, and blue, corresponding to four images of the same size). Assuming the convolution kernel size is 100*100, a total of 16 convolution kernels w1 to w16 are used. Kernel w1 to w4 form the first filter, kernels w5 to w8 form the second filter, kernels w9 to w12 form the third filter, and kernels w13 to w6 form the fourth filter. Different filters are used to extract different feature maps from the input image. Performing a convolution operation on the ARGB image using the first filter, that is, using w1 to w4 corresponding to the four images on the four channels, yields the first image. The top-left pixel of this image is the weighted sum of the pixels within a 100*100 area in the top-left corner of the four input images, and so on. Similarly, including the other filters, the output of this layer corresponds to four "images." Each image pair is a response to different features in the original image. The reason for extracting multiple feature maps in a convolutional layer is that we want to describe an image from multiple perspectives. Specifically, we use multiple different filters to convolve the image and obtain the responses on different filters as the image's feature data.

[0198] Step S1303: Calculate the first output data based on the attention vector corresponding to the target convolutional layer to obtain the second output data. The second output data includes N second feature maps that are robust to noise. The attention vector includes N elements, and each of the N elements is used to represent the robustness of the corresponding first feature map to noise.

[0199] Specifically, the attention vector is based on Figures 8 to 11 The updated first attention vector is obtained using the corresponding neural network training method. The second output data is calculated based on the attention vector corresponding to the target convolutional layer using the first output data. This can be implemented in software or based on... Figure 20 The hardware implementation described above. The output data of the aforementioned neural network system can be labels corresponding to the input data, such as video data, image data, or voice data. For example, if the input image data is a user's face image, then inputting the image data into the neural network model can output the user's identity information; similarly, if the input voice data is a user's voice, then inputting the voice data into the neural network model can output the user's identity information.

[0200] In existing technologies, during neural network computation, after receiving input data, the target convolutional layer performs convolution calculations on the input data based on its weights to obtain the output data. This output data is then directly used as the input data for the next layer. However, among all the feature maps (or feature data) in the output data of the target convolutional layer, the robustness of each feature map to noise varies. Some feature data are highly robust to noise, while others are poorly robust. Therefore, not all feature maps are suitable for direct input to the next layer. Therefore, this application proposes an attention vector to correct the output data obtained from weight-based convolution calculations. Specifically, in this embodiment, for any convolutional layer (such as a target convolutional layer) included in the neural network, in the process of obtaining the output data of the target convolutional layer, after performing convolution calculation on the input data based on the weights of the target convolutional layer to obtain the first output data, the first output data is further modified by the attention vector corresponding to the target convolutional layer. For example, feature maps with strong robustness to noise are strengthened, while feature maps with poor robustness to noise are weakened or even removed. The robustness of each feature data included in the extracted first output data to noise is comprehensively considered, and the best feature map is selected and input into the next layer for calculation of the next layer of the target convolutional layer, thereby reducing the impact of noise on the neural network system and improving the accuracy of neural network calculation.

[0201] In one possible implementation, the target convolutional layer includes M channels, and step S1304 may be included before obtaining the input data of the target convolutional layer.

[0202] Step S1304: Based on the attention vector, determine N channels from the M channels for performing the convolution calculation, wherein the element values ​​of the elements included in the attention vector are greater than or equal to a preset threshold, and M is an integer greater than N.

[0203] The target convolutional layer includes M channels, each channel corresponding to a weight set, and each channel is used to extract a feature map. Each channel corresponds to a weight set based on... Figures 8 to 11 The elements in the second attention vector obtained by training any of the corresponding method embodiments are, in this application embodiment, composed of elements in the trained second attention vector whose element values ​​are greater than or equal to a preset threshold.

[0204] Specifically, in the neural network computation process, before performing convolution calculations based on weights, channels (or filters) can be directly closed based on elements in the attention vector whose values ​​are less than a preset threshold. Only the remaining channels are used to extract noise-resistant feature data, thereby reducing the computational load of inference. For example, such as... Figure 14 As shown, Figure 14 This is a mathematical representation of a neural network computation method provided in this application embodiment. Taking image data as the input data as an example, the three matrices in the left region represent the input data of the original image in RGB format. The image data of the R, G, and B channels are represented by three matrices. The size of the original RGB format image is m×n×3 (i.e., width m, height n, and depth 3). Filter 1 represents the first filter, and Filter N represents the Nth filter. The filters have the same size, s×t×3 (i.e., height s, width t, and depth 3). Each filter consists of three convolutional kernels (three weights), and each convolutional kernel corresponds to one channel of the input data. Therefore, this convolution uses M (M=3×N) weights, i.e., N filters (corresponding to N weight sets: first weight set, second weight set, ..., Nth weight set). Thus, the target convolutional layer can extract N feature maps, and the depth of the output data of the target convolutional layer is N, i.e., the output data size of the target convolutional layer is k×j, and the depth is N. The size of the output data is related to the stride of the filter sliding. This convolutional layer corresponds to an attention vector p (p1, p2, p3, ..., pN), where element p1 corresponds to the first weight set (and the first filter), element p2 corresponds to the second weight set (and the second filter), and so on. When passing through Figures 8 to 11If an element in the final attention vector obtained by the corresponding neural network training method is less than a preset threshold, then, based on the attention vector obtained during training, the channel corresponding to the element whose value is less than the preset threshold can be closed. Based on the input data of the target convolutional layer, the attention vector (specifically, the element in the attention vector greater than the preset threshold), and the remaining channels, the output data of the target convolutional layer can be calculated, thereby simplifying the structure of the neural network and reducing the computational load of the neural network.

[0205] Specifically, since the attention vector filters the features extracted by the convolution matrix, optionally, the number of channels can be increased when extracting features to ensure that the filtered features have strong robustness to noise on the one hand, and that the number of extracted features is not reduced on the other hand, so that the extracted features can accurately describe the input data.

[0206] In this embodiment, for the target convolutional layer, N channels are determined from the M channels based on the attention vector to perform the convolution calculation. Feature maps with relatively strong noise robustness are extracted from these N channels. Simultaneously, different weighting coefficients are assigned to different feature maps based on their varying noise robustness, thereby correcting the feature maps. On one hand, reducing the number of channels decreases the computational load of the neural network inference, compressing the network model while maintaining computational accuracy. On the other hand, the attention vector can select the optimal feature map for the next layer of the target convolutional layer, preventing feature maps with poor noise robustness to components such as circuit devices from flowing into the next layer of the target convolutional layer. This reduces the impact of noise from circuit devices in the neural network system on the neural network system and improves the accuracy of neural network inference calculations.

[0207] In one possible implementation, a neural network training process is included before performing neural computation; specifically, the method further includes step S1305.

[0208] Step S1305: Adjust the initial network parameters of the neural network system based on the training data to obtain the trained network parameters of the neural network system. The adjustment process incorporates at least one of the simulated noise and actual noise of the circuit devices of the neural network system. The trained network parameters include the weights of the target convolutional layer and the attention vector corresponding to the target convolutional layer.

[0209] The process of training a neural network can be referenced here. Figures 8 to 11 The relevant descriptions of the corresponding neural network training methods will not be repeated here.

[0210] In one possible implementation, adjusting the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system includes:

[0211] The training data is input into the neural network system to obtain third output data; a loss value is calculated based on the third output data and the target output data; and the initial network parameters of the neural network are updated according to the loss value.

[0212] See Figure 15 , Figure 15 This is a flowchart illustrating a neural network training and calculation method provided in an embodiment of this application. The neural network training and calculation method includes the above-described... Figures 8 to 11 The neural network training method in any corresponding embodiment and as described above Figure 13 and Figure 14 The corresponding embodiment describes a neural network computation method. The implementation process of the neural network training method corresponds to the neural network training phase, and the implementation process of the neural network computation method corresponds to the neural network computation phase. The neural network training phase can also be called the software & hardware training phase, and the neural network computation phase can also be called the hardware inference phase. Figure 15 The corresponding neural network training and computation methods can be applied to Figure 16 On the corresponding chip system, among which, Figure 16 This is a schematic diagram of the structure of a chip system provided in an embodiment of this application.

[0213] I. The software & hardware training phase includes the following steps:

[0214] 1. Obtain training data and train L second weights and a second attention vector of the neural network model on the GPU based on the training data.

[0215] 2. Deploy the L second weights obtained from training on the GPU to... Figure 16 The corresponding memristor cross array of the chip system deploys the second attention vector into the digital storage module of the chip system.

[0216] 3. Based on the L second weights deployed on the memristor cross array of the chip system and the second attention vector of the digital storage module deployed on the chip system, combined with the actual noise of the memristor cross array, fix the L second weights deployed on the memristor cross array of the chip system, and retrain the second attention vector based on the training data to obtain the updated second attention vector.

[0217] II. The hardware inference phase includes the following steps:

[0218] 4. Redeploy the updated second attention vector obtained from the retraining to the digital storage module of the chip system;

[0219] 5. Control the opening and closing of each column of the memristor cross array according to the second attention vector;

[0220] 6. Data inference is performed based on L second weights deployed on the memristor cross array (the weights corresponding to the closed parts in the actual memristor cross array) and the second attention vector.

[0221] Its corresponding chip system architecture is as follows Figure 16 As shown. The above chip system may include: a switch control module, a memristor cross array, a digital storage module, and a multiplier, wherein each unit is described in detail below.

[0222] The memristor cross array is used to store L second weights for multiply-accumulate calculations based on the L second weight moments and the input data of the target convolutional layer.

[0223] The digital storage module is used to store the second attention vector, where each element of the second attention vector is used to determine the robustness of each feature data extracted by the second weight to noise.

[0224] The switch control module is used to control the closing of each column of the memristor cross array according to the second attention vector stored in the digital storage module, and to filter the memristor cross array columns participating in the calculation through switch control.

[0225] The multiplier is used for data inference based on L second weights deployed on the memristor cross array (actually the weights corresponding to the closed parts in the memristor cross array) and the second attention vector.

[0226] In this embodiment, the example of storing L second weights corresponding to the target convolutional layer using a memristor cross array and storing the second attention vector corresponding to the target convolutional layer using a digital storage module is provided. The memristor cross array can also be used to store weights corresponding to other convolutional layers of the neural network, and the digital storage module can also be used to store attention vectors corresponding to other convolutional layers of the neural network. This embodiment does not impose any limitations.

[0227] It is worth noting that, in addition to correcting the features extracted by the convolution matrix in the convolutional layer, the attention vector provided in this application embodiment can also correct the features of the output data after the convolutional / pooling layer in the fully connected layer. The correction process can be referred to the relevant description of the attention vector in the above embodiment, and will not be repeated in this application embodiment.

[0228] The methods of the embodiments of this application have been described in detail above, and the related apparatus of the embodiments of this application is provided below.

[0229] Please see Figure 17 , Figure 17 This is a schematic diagram of the structure of a neural network training device provided in an embodiment of this application. The neural network training device 17 may include an input unit 1701, a calculation unit 1702, and an update unit 1703, wherein each unit is described in detail below.

[0230] Input unit 1701 is used to acquire training data of the neural network system and obtain third output data; the target convolutional layer is any one of the at least one convolutional layer, and the target convolutional layer is used to calculate the input data of the target convolutional layer through a first attention vector and L first weights to obtain the input data of the next layer of the target convolutional layer; wherein, the L first weights are used for convolution calculation to extract feature data in the input data of the target convolutional layer, the first attention vector is used to correct the feature data, and L is an integer greater than or equal to 1;

[0231] The calculation unit 1702 is used to compare the third output data and the target output data to calculate the first loss value;

[0232] The update unit 1703 is used to update the network parameters of the neural network system according to the first loss value, the network parameters including the first attention vector and the L first weights.

[0233] In one possible implementation, the updated L first weights are L second weights, and the updated first attention vector is a second attention vector;

[0234] The input unit 1701 is further configured to input the training data into the neural network system to obtain fourth output data; the target convolutional layer calculates the input data of the target convolutional layer through the second attention vector and L third weights to obtain the input data of the next layer of the target convolutional layer; wherein, the L third weights correspond one-to-one with the L second weights, and each third weight is obtained by introducing the actual noise of the circuit devices in the neural network system after writing the corresponding second weight into the circuit devices of the neural network system;

[0235] The calculation unit 1702 is also used to compare the fourth output data and the target output data to calculate the second loss value;

[0236] The update unit 1703 is further configured to update the second attention vector in the network parameters according to the second loss value.

[0237] In one possible implementation, the apparatus further includes: a first processing unit 1704, configured to set elements in the initial attention vector that are less than or equal to the preset threshold to zero, while keeping elements in the vector that are greater than the preset threshold unchanged, to obtain the first attention vector.

[0238] In one possible implementation, the apparatus further includes: a first processing unit 1704, configured to process the initial attention vector based on an activation function; for the processed initial attention vector, setting elements in the vector that are less than or equal to the preset threshold to zero, while keeping elements in the vector that are greater than the preset threshold unchanged, to obtain the first attention vector.

[0239] In one possible implementation, the first processing unit 1704 is specifically used to: set elements in the vector less than or equal to the preset threshold to zero, and keep elements in the vector greater than the preset threshold unchanged to obtain the first attention vector;

[0240] The elements in the vector are sorted in descending order of value; the value of the Nth element after sorting is determined as the preset threshold, where N is determined based on a preset pruning rate, which represents the ratio of the number of invalid elements in the vector to the total number of elements in the vector; elements in the vector that are less than or equal to the preset threshold are set to zero, while elements in the vector that are greater than the preset threshold remain unchanged, thus obtaining the first attention vector.

[0241] In one possible implementation, the apparatus further includes a second processing unit 1705, configured to introduce read / write errors of circuit devices in a simulated neural network system into the L initial weights to obtain the L first weights, wherein the L initial weights correspond one-to-one with the L first weights. Optionally, the read / write errors of the circuit devices in the simulated neural network system follow a normal distribution, and the standard deviation of the normal distribution is correlated with the magnitude of the read / write errors of the circuit devices in the neural network system.

[0242] In one possible implementation, the updating unit 1703 is specifically configured to: update the initial attention vector and L initial weights according to the first loss value; update the L first weights based on the updated L initial weights; and update the first attention vector based on the updated initial attention vector.

[0243] It should be noted that the functions of each functional unit in the neural network training device 17 described in this application embodiment can be found in the relevant descriptions of each step in the above neural network training method embodiment, and will not be repeated here.

[0244] Please see Figure 18 , Figure 18 This is a schematic diagram of the structure of a neural network computing device 18 provided in an embodiment of this application. The neural network computing device 18 is applied to a neural network system including at least one convolutional layer and may include an acquisition unit 1801 and a computing unit 1802. The detailed description of each unit is as follows.

[0245] The acquisition unit 1801 is used to acquire the input data of the target convolutional layer, wherein the target convolutional layer is any one of the at least one convolutional layer;

[0246] The computing unit 1802 is used to perform convolution calculation on the input data based on the weights of the target convolutional layer to obtain first output data, wherein the first output data includes N first feature maps, where N is an integer greater than or equal to 1;

[0247] The computing unit 1802 is further configured to calculate the first output data based on the attention vector corresponding to the target convolutional layer to obtain second output data, wherein the second output data includes N second feature maps that are robust to noise, and the attention vector includes N elements, each of the N elements being used to represent the robustness of the corresponding first feature data to noise.

[0248] In one possible implementation, the target convolutional layer includes M channels, and the apparatus further includes a determination unit 1803.

[0249] The determining unit 1803 is used to determine N channels for performing the convolution calculation from the M channels based on the attention vector, wherein the element values ​​of the elements included in the attention vector are greater than or equal to a preset threshold, and M is an integer greater than N.

[0250] In one possible implementation, the device further includes a training unit 1804.

[0251] The training unit 1804 is used to adjust the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system. The adjustment process incorporates at least one of the simulated noise and actual noise of the circuit devices of the neural network system. The trained network parameters include the weights of the target convolutional layer and the attention vector corresponding to the target convolutional layer.

[0252] In one possible implementation, when adjusting the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system, the training unit 1804 is specifically used for:

[0253] The training data is input into the neural network system to obtain third output data; a loss value is calculated based on the third output data and the target output data; and the initial network parameters of the neural network are updated according to the loss value.

[0254] The weights are based on Figures 8 to 11 The updated L first weights obtained by the corresponding neural network training method, wherein the attention vector is based on... Figures 8 to 11 The updated first attention vector is determined by the corresponding neural network training method.

[0255] It should be noted that the functions of each functional unit in the neural network computing device 18 described in the embodiments of this application can be found in the relevant descriptions of each step in the above-mentioned neural network computing method embodiments, and will not be repeated here.

[0256] Figure 19 This is a schematic diagram of another neural network training device provided in an embodiment of this application, as shown below. Figure 19 As shown, the neural network training device 19 includes at least one processor 191, at least one memory 192, and at least one communication interface 193. In addition, the device may also include general-purpose components such as antennas, which will not be detailed here.

[0257] Processor 191 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs in the above scheme.

[0258] Communication interface 193 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), core network, Wireless Local Area Networks (WLAN), etc.

[0259] Memory 192 may be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processor via a bus. Memory may also be integrated with the processor.

[0260] The memory 192 stores the application code for executing the above scheme, and its execution is controlled by the processor 191. The processor 191 executes the application code stored in the memory 192.

[0261] The code stored in memory 192 can execute the above. Figures 8 to 11 The provided neural network training method includes, for example, acquiring training data of the neural network system to obtain third output data; the target convolutional layer is any one of the at least one convolutional layers, and the target convolutional layer is used to calculate the input data of the target convolutional layer through a first attention vector and L first weights to obtain the input data of the next layer of the target convolutional layer; wherein, the L first weights are used for convolution calculation to extract feature data from the input data of the target convolutional layer, the first attention vector is used to correct the feature data, and L is an integer greater than or equal to 1; the third output data and the target output data are compared to calculate a first loss value; the network parameters of the neural network system are updated according to the first loss value, and the network parameters include the first attention vector and the L first weights.

[0262] It should be noted that the functions of each functional unit in the neural network training device described in the embodiments of this application can be found in the relevant descriptions of each step in the above-mentioned neural network training method embodiments, and will not be repeated here.

[0263] Figure 20 This is a schematic diagram of another neural network computing device provided in an embodiment of this application, as shown below. Figure 20As shown, the neural network computing device 20 may include a first storage unit 2001, a second storage unit 2002, a third storage unit 2003, and a processing unit 2004, wherein each unit is described in detail below.

[0264] The first storage unit 2001 is used to store the input data of a neural network system. The neural network system includes at least one convolutional layer. A target convolutional layer is used to calculate the input data of the next layer of the target convolutional layer by using a second attention vector and L second weights. The L second weights are used for convolution calculation to extract feature data from the input data of the target convolutional layer. The second attention vector is used to correct the feature data. The network parameters include the second attention vector and the L second weights. The target convolutional layer is any one of the at least one convolutional layer.

[0265] The second storage unit 2002 is used to store the L second weights.

[0266] The third storage unit 2003 is used to store the second attention vector.

[0267] The processing unit 2004 is used to acquire the input data and network parameters, and calculate the output data of the neural network computing device based on the input data and the network parameters, wherein the network parameters include the second attention vector and the L second weights.

[0268] Wherein, the L second weights are based on Figures 8 to 11 The corresponding neural network training method obtains the updated L first weights, and the second attention vector is based on... Figures 8 to 11 The updated first attention vector obtained by the corresponding neural network training method.

[0269] In one possible implementation, the device further includes a switch controller 2005 and M switches (first switch 2006-1, second switch 2006-2, ..., Mth switch 2006-M), the second storage unit includes M storage sub-units (first storage sub-unit 2002-1, second storage sub-unit 2002-2, ..., Mth storage sub-unit 2002-M), the M storage sub-units are connected one-to-one with the M switches, the switch controller 2005 is connected to the M switches, and the switch controller 2005 is connected to the third storage unit 2003.

[0270] The L second weights include M weight sets, and the union of the M weight sets is the L second weights; wherein, one weight set is used for convolution calculation to extract a feature map of the input data of the target convolutional layer, each of the M storage sub-units is used to store one weight set, and the second attention vector includes M elements, and there is a one-to-one correspondence between the M elements, the M switches and the M weight sets;

[0271] The switch controller 2005 is configured to control the switch corresponding to the target element to open when the value of the target element in the second attention vector is greater than or equal to a preset threshold, so that the processing unit can obtain the weight set corresponding to the value of the target element being greater than or equal to the preset threshold, and to control the switch corresponding to the target element to close when the value of an element in the second attention vector is less than or equal to the preset threshold, wherein the target element is any element in the second attention vector;

[0272] The processing unit 2004 is further configured to: determine N weight sets based on the values ​​of the M elements and the M weight sets, wherein the N weight sets are weight sets corresponding to elements in the M weight sets whose element values ​​are greater than or equal to a preset threshold; determine a third attention vector based on the values ​​of the M elements and the second attention vector, wherein the third attention vector is composed of elements in the second attention vector whose element values ​​are greater than or equal to a preset threshold; wherein the target convolutional layer is configured to calculate the input data of the target convolutional layer using the second attention vector and L second weights to obtain the input data of the next layer of the target convolutional layer, including: the target convolutional layer is configured to calculate the input data of the target convolutional layer using the third attention vector and the N weight sets to obtain the input data of the next layer of the target convolutional layer.

[0273] The aforementioned processing unit 2004 may be a processor, such as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the above scheme program.

[0274] The first storage unit 2001, the second storage unit 2002, and the third storage unit 2003 mentioned above can be memory, such as read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0275] Optionally, the neural network computing device in this application embodiment may further include a communication interface for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), core network, Wireless Local Area Network (WLAN), etc.

[0276] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0277] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0278] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0279] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0280] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0281] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0282] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A neural network computation method, characterized in that, The method is performed by a neural network system including at least one convolutional layer, the neural network system running on a hardware platform containing circuit devices, the method comprising: Obtain input data for the target convolutional layer, wherein the target convolutional layer is any one of the at least one convolutional layers; the target convolutional layer includes M channels; N channels for performing convolution calculations are determined from the M channels based on the attention vector corresponding to the target convolutional layer, where M is an integer greater than N; Based on the weights of the target convolutional layer, the input data is convolved through the N channels to obtain first output data, wherein the first output data includes N first feature maps, where N is an integer greater than or equal to 1; Based on the attention vector, the first output data is calculated through the N channels to obtain the second output data, wherein the second output data includes N second feature maps that are robust to the noise of the circuit device; The attention vector includes N elements, each of which has an element value greater than or equal to a preset threshold. These elements represent the robustness of the corresponding feature map to noise in the circuit device, with larger element values ​​indicating stronger robustness of the feature data to noise.

2. The method according to claim 1, characterized in that, The method further includes: The initial network parameters of the neural network system are adjusted based on the training data to obtain the trained network parameters of the neural network system. The adjustment process incorporates at least one of the simulated noise and actual noise of the circuit devices of the neural network system. The trained network parameters include the weights of the target convolutional layer and the attention vector corresponding to the target convolutional layer.

3. The method according to claim 2, characterized in that, The step of adjusting the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system includes: The training data is input into the neural network system to obtain the third output data; The loss value is obtained by calculating based on the third output data and the target output data; The initial network parameters of the neural network are updated based on the loss value.

4. A neural network computing device, characterized in that, The apparatus is applied to a neural network system including at least one convolutional layer, the neural network system running on a hardware platform containing circuit devices, the apparatus comprising: An acquisition unit is used to acquire input data of a target convolutional layer, wherein the target convolutional layer is any one of the at least one convolutional layers; the target convolutional layer includes M channels; The determining unit is used to determine N channels for performing convolution calculations from the M channels based on the attention vector corresponding to the target convolutional layer, where M is an integer greater than N; The computing unit is used to perform convolution calculation on the input data through the N channels based on the weights of the target convolutional layer to obtain first output data, wherein the first output data includes N first feature maps, and N is an integer greater than or equal to 1; The computing unit is further configured to calculate the first output data based on the attention vector through the N channels to obtain second output data, wherein the second output data includes N second feature maps that are robust to the noise of the circuit device; The attention vector includes N elements, each of which has an element value greater than or equal to a preset threshold. These elements represent the robustness of the corresponding feature map to noise in the circuit device, with larger element values ​​indicating stronger robustness of the feature data to noise.

5. The apparatus according to claim 4, characterized in that, The device further includes: A training unit is used to adjust the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system. The adjustment process incorporates at least one of the simulated noise and actual noise of the circuit devices of the neural network system. The trained network parameters include the weights of the target convolutional layer and the attention vector corresponding to the target convolutional layer.

6. The apparatus according to claim 5, characterized in that, When adjusting the initial network parameters of the neural network system based on training data to obtain the trained network parameters of the neural network system, the training unit is specifically used for: The training data is input into the neural network system to obtain third output data; a loss value is calculated based on the third output data and the target output data; and the initial network parameters of the neural network are updated according to the loss value.

7. A neural network computing device, characterized in that, The device includes a processor, a memory, and a communication interface, wherein the memory is used to store neural network calculation program code, and the processor is used to call the neural network calculation program code to execute the method of any one of claims 1 to 3.

8. A chip system, characterized in that, The chip system includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected by a circuit. The at least one memory stores instructions. When the instructions are executed by the processor, the neural network computing method according to any one of claims 1 to 3 is implemented.

9. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the neural network calculation method described in any one of claims 1 to 3.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computer, cause the computer to perform the neural network computation method according to any one of claims 1 to 3.

Images